MarcoWel commented 1 year ago

When setting up the Deutsche Post DHL carrier correctly, there will be the following errors:

Tracking: "Log-in failed"

Label creation: Encoding error

MarcoWel commented 1 year ago

Issue persists in 2023.1.8: karrio seems to sucessfully send the request but receives a 500 error back which might not be properly handled:

karrio.api   | DEBUG helpers.py 189 sending request (b00523e2-67cd-436c-aa8c-d5ea9a547a1f)...
karrio.api   | INFO helpers.py 130 Request URL:: https://cig.dhl.de/services/production/soap
karrio.api   | ERROR helpers.py 162 HTTP Error 500: Server Error
karrio.api   | ERROR interface.py 64 'utf-8' codec can't decode byte 0xe4 in position 316: invalid continuation byte

MarcoWel commented 1 year ago

DPDHL still does not work for us with live credentials.

DPDHL needs App ID + Token for Basic Auth first, then business portal User + Password in the XML request in the SOAP AUthentification header.
Karrio should show HTTP errors clearly in the request logs instead of a generic error message
Karrio should show a distinct error when Basic Auth fails with 401, e.g. "DPDHL App ID / Token incorrect"
DPDHL HTTP 500 error after passing basic auth with live credentials needs to be resolved

Request Log: "Document is empty, line 1, column 1 (<string>, line 1)" Docker Log:

karrio.api   | DEBUG helpers.py 189 sending request (7ce9b009-f4a6-48c8-bc54-9a8ea4b9f565)...
karrio.api   | INFO helpers.py 130 Request URL:: https://cig.dhl.de/services/production/soap
karrio.api   | ERROR helpers.py 162 HTTP Error 500: Server Error
karrio.api   | DEBUG helpers.py 172 Error content:: HTTP Error 500: Server Error
karrio.api   | DEBUG serializable.py 30 
karrio.api   | ERROR interface.py 64 Document is empty, line 1, column 1 (<string>, line 1)

Request Log: {"code":"failure"} Docker Log:

karrio.api   | DEBUG helpers.py 189 sending request (7d90eeb0-3ac6-4de9-b244-de72fa05388f)...
karrio.api   | INFO helpers.py 130 Request URL:: https://cig.dhl.de/services/production/soap
karrio.api   | ERROR helpers.py 162 HTTP Error 401: Unauthorized
karrio.api   | DEBUG helpers.py 172 Error content:: HTTP Error 401: Unauthorized
karrio.api   | DEBUG serializable.py 30 <data code="401" error="Unauthorized">
karrio.api   |             <Status>
karrio.api   |                 <statusCode>401</statusCode>
karrio.api   |                 <statusText>Unauthorized</statusText>
karrio.api   |             </Status>
karrio.api   |         </data>

danh91 commented 1 year ago

The authentication in production is still returning Unauthorized error

MarcoWel commented 1 year ago

Tested with 2023.1.10, unfortunately it is still not working. The displayed errors are still the same generic ones, but the Docker logs reveal slightly more now.

Wrong basic auth (App ID/password): Still generic error [ { "code": "failure" } ], no details in the dev log. Docker log showing:

karrio.api        | DEBUG helpers.py 195 sending request (60affd63-2a49-4206-9fab-4c6eb0d38f71)...
karrio.api        | INFO helpers.py 136 Request URL:: https://cig.dhl.de/services/production/soap
karrio.api        | ERROR helpers.py 168 HTTP Error 401: Unauthorized
karrio.api        | DEBUG helpers.py 178 Error content:: HTTP Error 401: Unauthorized
karrio.api        | DEBUG serializable.py 30 <data code="401" error="Unauthorized">
karrio.api        |             <Status>
karrio.api        |                 <statusCode>401</statusCode>
karrio.api        |                 <statusText>Unauthorized</statusText>
karrio.api        |             </Status>
karrio.api        |         </data>

With correct basic auth and credentials (user/signature): Still getting generic error [ { "carrier_id": "DHL", "carrier_name": "dpdhl", "code": 4, "message": "Document is empty, line 1, column 1 (<string>, line 1)" } ], no details in the dev log. Docker log showing:

karrio.api        | DEBUG helpers.py 195 sending request (e77211dc-650e-4fb0-a6b0-97aa0b8568bc)...
karrio.api        | INFO helpers.py 136 Request URL:: https://cig.dhl.de/services/production/soap
karrio.api        | ERROR helpers.py 168 HTTP Error 500: Server Error
karrio.api        | DEBUG helpers.py 178 Error content:: HTTP Error 500: Server Error
karrio.api        | DEBUG serializable.py 30 
karrio.api        | ERROR interface.py 64 Document is empty, line 1, column 1 (<string>, line 1)

XML request still seems to have differing schema version (3.0) from stated request version (3.4):

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:cis="http://dhl.de/webservice/cisbase" xmlns:ns="http://dhl.de/webservices/businesscustomershipping/3.0">
    <soapenv:Header>
        <cis:Authentification>
            <cis:user>...</cis:user>
            <cis:signature>...</cis:signature>
        </cis:Authentification>
    </soapenv:Header>
    <soapenv:Body>
        <ns:CreateShipmentOrderRequest>
            <ns:Version>
                <majorRelease>3</majorRelease>
                <minorRelease>4</minorRelease>
            </ns:Version>
            ...

MarcoWel commented 1 year ago

The behavior is the same for production (as shown above) and sandbox (in test mode), where the request looks as follows:

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:cis="http://dhl.de/webservice/cisbase" xmlns:ns="http://dhl.de/webservices/businesscustomershipping/3.0">
    <soapenv:Header>
        <cis:Authentification>
            <cis:user>2222222222_01</cis:user>
            <cis:signature>pass</cis:signature>
        </cis:Authentification>
    </soapenv:Header>
    <soapenv:Body>
        <ns:CreateShipmentOrderRequest>
            <ns:Version>
                <majorRelease>3</majorRelease>
                <minorRelease>4</minorRelease>
            </ns:Version>

MarcoWel commented 1 year ago

Server error 500 is caused by missing postfix in account number, see #333.

Please close this issue once the generic server errors are translated to a proper error (e.g. "Site-ID or Signature wrong" for error 401, "Check Account Number " for error 500).

MarcoWel commented 1 year ago

Update: Label creation working, tracking still failing with original error of this issue "Log-in failed" for both sandbox and production:

MarcoWel commented 1 year ago

It seems that for tracking in test mode, DHL expects the dev user ID + password in the xml part of the request, not the app-id + password. Swapping the credentials in karrio leads to the request looking correct, but results in a 401 Unauthorized error.

https://entwickler.dhl.de/group/ep/wsapis/sendungsverfolgung/authentifizierung

MarcoWel commented 1 year ago

Weirdly, for tracking HTTP errors are reported in the dashboard logs, but they are still not appearing under request errors, plus the request is listed as "successful". For label creation, HTTP errors are not reported anywhere but the request is listed under "failed".

danh91 commented 1 year ago

Hi @MarcoWel ,

I am hoping to get this fixed in the next patch. Could you please send me a concrete sample of the requests that you got working on your end:

I added the code to handle the postfix as you mentioned https://github.com/karrioapi/karrio/issues/333 based on https://entwickler.dhl.de/group/ep/wsapis/geschaeftskundenversand/current/authentifizierung

What Is Missing

Tracking

Test mode

Basic Auth: username:password => [DEVELOPER_ID]:[DEV_PORTAL_PASSWORD] +

<?xml version="1.0" encoding="UTF-8" standalone="no"?> 
<data appname="zt12345" 
            language-code="en" 
            password="geheim" 
            piece-code="00340434161094015902"
            request="d-get-piece-detail"/>

This works ^

Production

Basic Auth: username:password => [APP_ID]:[APP_TOKEN] +

<?xml version="1.0" encoding="UTF-8" standalone="no"?> 
<data appname="[WHAT_GOES_HERE?]" 
            language-code="en" 
            password="[WHAT_GOES_HERE?]" 
            piece-code="00340434161094015902"
            request="d-get-piece-detail"/>

Shipment

Test mode

Basic Auth: username:password => [DEVELOPER_ID]:[DEV_PORTAL_PASSWORD] +

<soapenv:Envelope
    xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"
    xmlns:cis="http://dhl.de/webservice/cisbase"
    xmlns:ns="http://dhl.de/webservices/businesscustomershipping/3.0">
    <soapenv:Header>
        <cis:Authentification>
            <cis:user>2222222222_01</cis:user>
            <cis:signature>pass</cis:signature>
        </cis:Authentification>
    </soapenv:Header>
    <soapenv:Body>
        <ns:CreateShipmentOrderRequest>
            <ns:Version>
                <majorRelease>3</majorRelease>
                <minorRelease>4</minorRelease>
            </ns:Version>
            <ShipmentOrder>
                <sequenceNumber>1</sequenceNumber>
                <Shipment>
                    <ShipmentDetails>
                        <product>V01PAK</product>
                        <cis:accountNumber>22222222220102</cis:accountNumber>
                                                <ShipmentItem>
                            <weightInKG>1</weightInKG>
                            <lengthInCM>10</lengthInCM>
                            <widthInCM>33</widthInCM>
                            <heightInCM>18</heightInCM>
                        </ShipmentItem>
                        <Notification/>
                    </ShipmentDetails>
                    <Shipper>
                        <Name>
                            <cis:name1>Deut</cis:name1>
                            <cis:name2></cis:name2>
                        </Name>
                        <Address>
                            <cis:streetName>Römerberg</cis:streetName>
                            <cis:streetNumber>23</cis:streetNumber>
                            <cis:zip>60311</cis:zip>
                            <cis:city>Frankfurt</cis:city>
                            <cis:Origin>
                                <cis:country>Germany</cis:country>
                                <cis:countryISOCode>DE</cis:countryISOCode>
                            </cis:Origin>
                        </Address>
                        <Communication>
                            <cis:contactPerson>Deut</cis:contactPerson>
                        </Communication>
                    </Shipper>
                    <ShipperReference></ShipperReference>
                    <Receiver>
                        <cis:name1>Ana</cis:name1>
                        <Address>
                            <cis:streetName>Am Wall</cis:streetName>
                            <cis:streetNumber>207</cis:streetNumber>
                            <cis:zip>28195</cis:zip>
                            <cis:city>Bremen</cis:city>
                            <cis:Origin>
                                <cis:country>Germany</cis:country>
                                <cis:countryISOCode>DE</cis:countryISOCode>
                            </cis:Origin>
                        </Address>
                        <Communication>
                            <cis:contactPerson>Ana</cis:contactPerson>
                        </Communication>
                    </Receiver>
                </Shipment>
            </ShipmentOrder>
            <labelResponseType>B64</labelResponseType>
            <combinedPrinting>0</combinedPrinting>
        </ns:CreateShipmentOrderRequest>
    </soapenv:Body>
</soapenv:Envelope>

This works ^

Production

Basic Auth: username:password => [APP_ID]:[APP_TOKEN] +

<soapenv:Envelope
    xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"
    xmlns:cis="http://dhl.de/webservice/cisbase"
    xmlns:ns="http://dhl.de/webservices/businesscustomershipping/3.0">
    <soapenv:Header>
        <cis:Authentification>
            <cis:user>[EKP]_01</cis:user>
            <cis:signature>[WHAT_GOES_HERE?]</cis:signature>
        </cis:Authentification>
    </soapenv:Header>
    <soapenv:Body>
        <ns:CreateShipmentOrderRequest>
            <ns:Version>
                <majorRelease>3</majorRelease>
                <minorRelease>4</minorRelease>
            </ns:Version>
            <ShipmentOrder>
                <sequenceNumber>1</sequenceNumber>
                <Shipment>
                    <ShipmentDetails>
                        <product>V01PAK</product>
                        <cis:accountNumber>[EKP]0102</cis:accountNumber>

The challenge is I don't have an EKP (DHL customer number) so I am unable to test the requests production If you can sen me working request with clear indication of the [WHAT_GOES_HERE?] (developer id? app id?...) indications, it would greatly help. Thanks

danh91 commented 1 year ago

Weirdly, for tracking HTTP errors are reported in the dashboard logs, but they are still not appearing under request errors, plus the request is listed as "successful". For label creation, HTTP errors are not reported anywhere but the request is listed under "failed".

This is by design. The idea is to create the tracker whether the tracking number returns a valid response or not. The use case is when you create or import shipments into Karrio that haven't been picked up by carriers yet, the tracking number might not be in the system yet but will be soon.

I agree that it might create confusion though. I am thinking of adding an API parameter to make that explicit. So basically:

POST /v1/trackers with { tracking_number: 000, carrier_name: ___ } will return an error if tracking fails
POST /v1/trackers with { tracking_number: 000, carrier_name: ___, pending_pickup: true } will create the tracker even if tracking fails

Let me know what you think

MarcoWel commented 1 year ago

For DHL, tracking should work immediately after label creation (so there should not be an HTTP error). In any case, HTTP errors should be properly displayed in the karrio logs IMO.

Please note the following:

EKP = DHL account number (every DHL customer has at least one)
GKP = Geschäftskundenportal = Business Customer Portal. Every DHL customer has a business portal user and can add more users, e.g. a dedicated one for karrio.
ZT = ZT_ID / Password is used for DHL tracking only and needs to be requested by the user at the DHL customer service (DHL Universal is the newer standard for tracking and can be used instead).

Let me add in the correct bits for production.

Shipment

Basic Auth: username:password => [APP_ID]:[APP_TOKEN]

<soapenv:Envelope
    xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"
    xmlns:cis="http://dhl.de/webservice/cisbase"
    xmlns:ns="http://dhl.de/webservices/businesscustomershipping/3.0">
    <soapenv:Header>
        <cis:Authentification>
            <cis:user>[GKP_Username]</cis:user>
            <cis:signature>[GKP_Password]</cis:signature>
        </cis:Authentification>
    </soapenv:Header>
    <soapenv:Body>
        <ns:CreateShipmentOrderRequest>
            <ns:Version>
                <majorRelease>3</majorRelease>
                <minorRelease>4</minorRelease>
            </ns:Version>
            <ShipmentOrder>
                <sequenceNumber>1</sequenceNumber>
                <Shipment>
                    <ShipmentDetails>
                        <product>V01PAK</product>
                        <cis:accountNumber>[EKP]0102</cis:accountNumber>

Tracking

Basic Auth: username:password => [APP_ID]:[APP_TOKEN]

<?xml version="1.0" encoding="UTF-8" standalone="no"?> 
<data appname="[ZT_ID]" 
            password="[ZT_Password]" 
            language-code="en" 
            piece-code="0034xxxxxxxxxxx"
            request="d-get-piece-detail"/>

Ref.: https://shipxpert.info/dhl-de/

MarcoWel commented 1 year ago

Tested with 2022.1.12.

Findings for Shipment are documented in #333.
For tracking, we got our ZT-ID from DHL. Tracking auth works, but now there is a 400 "Bad request" when trying to add a DPDHL tracker.

The request looks like this:

<data appname="[ZT_ID]" 
            password="[ZT_Password]" 
            language-code="en" 
            piece-code="0034xxxxxxxxxxx"
            request="d-get-piece-detail"/>

Could it be the header line is missing like follows?

<?xml version="1.0" encoding="UTF-8" standalone="no"?> 
<data appname="[ZT_ID]" 
            password="[ZT_Password]" 
            language-code="en" 
            piece-code="0034xxxxxxxxxxx"
            request="d-get-piece-detail"/>

MarcoWel commented 1 year ago

@danh91 Tested with 2023.3. There seems to be a new bug so DPDHL is currently not working at all. karrio does not even get to send a request to DPDHL but throws an error before:

karrio.api        | ERROR interface.py 64 'dict' object has no attribute 'cities'
karrio.api        | Traceback (most recent call last):
karrio.api        |   File "/karrio/venv/lib/python3.10/site-packages/karrio/api/interface.py", line 62, in wrapper
karrio.api        |     return func(*args, **kwargs)
karrio.api        |   File "/karrio/venv/lib/python3.10/site-packages/karrio/api/interface.py", line 279, in process
karrio.api        |     response: lib.Deserializable = gateway.proxy.get_rates(request)
karrio.api        |   File "/karrio/venv/lib/python3.10/site-packages/karrio/mappers/dpdhl/proxy.py", line 17, in get_rates
karrio.api        |     return super().get_rates(request)
karrio.api        |   File "/karrio/venv/lib/python3.10/site-packages/karrio/universal/mappers/rating_proxy.py", line 48, in get_rates
karrio.api        |     response: typing.List[typing.Tuple[str, PackageRates]] = [
karrio.api        |   File "/karrio/venv/lib/python3.10/site-packages/karrio/universal/mappers/rating_proxy.py", line 51, in <listcomp>
karrio.api        |     get_available_rates(
karrio.api        |   File "/karrio/venv/lib/python3.10/site-packages/karrio/universal/mappers/rating_proxy.py", line 146, in get_available_rates
karrio.api        |     _cover_supported_cities = recipient.city in zone.cities or not any(
karrio.api        | AttributeError: 'dict' object has no attribute 'cities'

danh91 commented 1 year ago

@danh91 Tested with 2023.3. There seems to be a new bug so DPDHL is currently not working at all. karrio does not even get to send a request to DPDHL but throws an error before:

karrio.api        | ERROR interface.py 64 'dict' object has no attribute 'cities'
karrio.api        | Traceback (most recent call last):
karrio.api        |   File "/karrio/venv/lib/python3.10/site-packages/karrio/api/interface.py", line 62, in wrapper
karrio.api        |     return func(*args, **kwargs)
karrio.api        |   File "/karrio/venv/lib/python3.10/site-packages/karrio/api/interface.py", line 279, in process
karrio.api        |     response: lib.Deserializable = gateway.proxy.get_rates(request)
karrio.api        |   File "/karrio/venv/lib/python3.10/site-packages/karrio/mappers/dpdhl/proxy.py", line 17, in get_rates
karrio.api        |     return super().get_rates(request)
karrio.api        |   File "/karrio/venv/lib/python3.10/site-packages/karrio/universal/mappers/rating_proxy.py", line 48, in get_rates
karrio.api        |     response: typing.List[typing.Tuple[str, PackageRates]] = [
karrio.api        |   File "/karrio/venv/lib/python3.10/site-packages/karrio/universal/mappers/rating_proxy.py", line 51, in <listcomp>
karrio.api        |     get_available_rates(
karrio.api        |   File "/karrio/venv/lib/python3.10/site-packages/karrio/universal/mappers/rating_proxy.py", line 146, in get_available_rates
karrio.api        |     _cover_supported_cities = recipient.city in zone.cities or not any(
karrio.api        | AttributeError: 'dict' object has no attribute 'cities'

Benign changes at the last minute introduced this bug. I just pushed a hot-fix 2023.3.01

danh91 commented 1 year ago

Tested with 2022.1.12.

Findings for Shipment are documented in DPDHL: Add postfix to account number depending on product #333.

For tracking, we got our ZT-ID from DHL. Tracking auth works, but now there is a 400 "Bad request" when trying to add a DPDHL tracker.

The request looks like this:
<data appname="[ZT_ID]" 
          password="[ZT_Password]" 
          language-code="en" 
          piece-code="0034xxxxxxxxxxx"
          request="d-get-piece-detail"/> 
Could it be the header line is missing like follows?
<?xml version="1.0" encoding="UTF-8" standalone="no"?> 
<data appname="[ZT_ID]" 
          password="[ZT_Password]" 
          language-code="en" 
          piece-code="0034xxxxxxxxxxx"
          request="d-get-piece-detail"/> 

Can you confirm whether your ZT credentials work when you make the request with insomnia? From the tests we did <?xml version="1.0" encoding="UTF-8" standalone="no"?> didn't make a difference for the tracking request in test mode

MarcoWel commented 1 year ago

It seems to be an issue on DHL API side. Sandbox works flawlessly, the production endpoint is giving 400 responses no matter what (even with empty body). We opened a support ticket with DHL.

One improvement suggestion for karrio would be to display the full error details:

MarcoWel commented 1 year ago

Got the DPDHL tracking to work now.

The request can be sent as follows:

GET: (Basic Auth: [APP_ID]:[APP_TOKEN])

https://cig.dhl.de/services/production/rest/sendungsverfolgung?xml=<?xml version="1.0" encoding="UTF-8" standalone="no"?> <data appname="zt000000" language-code="en" password="xxxxxx" piece-code="0034xxxxxxxxxxxxxxxx" request="d-get-piece-detail"/>

POST: (Basic Auth: [APP_ID]:[APP_TOKEN])

curl --location 'https://cig.dhl.de/services/production/rest/sendungsverfolgung' \
--header 'Content-Type: application/x-www-form-urlencoded' \
--header 'Authorization: Basic xxxxxxxxxxxxxxxxx' \
--data-urlencode 'xml=<?xml version="1.0" encoding="UTF-8" standalone="no"?> <data appname="zt000000" language-code="en" password="xxxxxx" piece-code="0034xxxxxxxxxxxxxxxx" request="d-get-piece-detail"/>'

MarcoWel commented 1 year ago

Both production AND sandbox endpoints work well with above shown solution.

Strangely, the sandbox endpoint behaves differently and works well with the request sent by karrio (xml data as raw in POST body). The production endpoint throws a 400 error with the same request (yet adapted basic auth credentials).

danh91 commented 1 year ago

So basically it needs the xml= to work in both cases 😓

danh91 commented 1 year ago

I test

Got the DPDHL tracking to work now.

The request can be sent as follows:

GET: (Basic Auth: [APP_ID]:[APP_TOKEN])

https://cig.dhl.de/services/production/rest/sendungsverfolgung?xml=<?xml version="1.0" encoding="UTF-8" standalone="no"?> <data appname="zt000000" language-code="en" password="xxxxxx" piece-code="0034xxxxxxxxxxxxxxxx" request="d-get-piece-detail"/>

POST: (Basic Auth: [APP_ID]:[APP_TOKEN])

curl --location 'https://cig.dhl.de/services/production/rest/sendungsverfolgung' \
--header 'Content-Type: application/x-www-form-urlencoded' \
--header 'Authorization: Basic xxxxxxxxxxxxxxxxx' \
--data-urlencode 'xml=<?xml version="1.0" encoding="UTF-8" standalone="no"?> <data appname="zt000000" language-code="en" password="xxxxxx" piece-code="0034xxxxxxxxxxxxxxxx" request="d-get-piece-detail"/>'

I updated the code to use this approach and tested it with sandbox. It seems to work well.

Thanks

MarcoWel commented 1 year ago

@danh91 Tested with 2023.4.1.01. Confirm it still works with sandbox. For production, the request itself works now (getting the tracking data back from DPDHL in the response), but there is a parsing error now:

{
  "messages": [
    {
      "carrier_id": "DHL",
      "carrier_name": "dpdhl",
      "code": "SHIPPING_SDK_INTERNAL_ERROR",
      "message": "Requires integer value: invalid literal for int() with base 10: '' (element data/line 2)"
    }
  ]
}

MarcoWel commented 1 year ago

I can confirm that DPDHL tracking now finally works with 2023.4.2 🥳

MarcoWel commented 8 months ago

Since DPDHL API migration, tracking is giving the following error (tested with karrio 2024.2.rc1)

{
    "status": 401,
    "title": "Unauthorized",
    "detail": "Unauthorized for given resource."
}

Solution is to use GET request instead of POST (see here).

danh91 commented 8 months ago

Since DPDHL API migration, tracking is giving the following error (tested with karrio 2024.2.rc1)
{
    "status": 401,
    "title": "Unauthorized",
    "detail": "Unauthorized for given resource."
}
Solution is to use GET request instead of POST (see here).

Hi @MarcoWel, I vaguely remember there was an issue with GET. What version did you upgrade from? Was it working on that version?

MarcoWel commented 8 months ago

Upgraded from 2023.5 and it worked back then - did not test 2023.5 after API migration happened on DHL side though. It seems they shut down the POST endpoint and only allow GET requests now.

karrioapi / karrio

DPDHL: Tracking / Label creation not working #314

What Is Missing

Tracking

Test mode

Production

Shipment

Test mode

Production