[Importer] The Inventory upload template seems to break

dubdabasoduba commented 2 weeks ago

Context

The Inventory location assignment is based on the LIST resource used as linkage.
This seems to break when search for the LIST before creating the inventory

Errors display

progress::Building payload   [####################################]  100%
Posting request
Request type: POST
Url: https://unicef-mg-eusm-preview.smartregister.org/fhir
Progress::Linking inventory to location  [####################################]  100%Posting request
Request type: GET
Url: https://unicef-mg-eusm-preview.smartregister.org/fhir/List/_search?_count=1&subject=Location/e01047f8-c160-4ff7-8a8d-d46eb1968f95
[200]: SUCCESS!

Traceback (most recent call last):
  File "/Users/dubdabasoduba/code/ona/server/fhir-tooling/importer/main.py", line 1961, in <module>
    main()
  File "/Users/dubdabasoduba/code/ona/virtual-envs/lib/python3.12/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dubdabasoduba/code/ona/virtual-envs/lib/python3.12/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/Users/dubdabasoduba/code/ona/virtual-envs/lib/python3.12/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dubdabasoduba/code/ona/virtual-envs/lib/python3.12/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dubdabasoduba/code/ona/server/fhir-tooling/importer/main.py", line 1935, in main
    link_payload = link_to_location(resource_list)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dubdabasoduba/code/ona/server/fhir-tooling/importer/main.py", line 1093, in link_to_location
    return build_assign_payload(arr, "List", "subject=Location/")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dubdabasoduba/code/ona/server/fhir-tooling/importer/main.py", line 851, in build_assign_payload
    get_content = check_resource(subject_id, entries, resource_type, url_filter)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dubdabasoduba/code/ona/server/fhir-tooling/importer/main.py", line 826, in check_resource
    json_response = json.loads(response[0])
                    ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Implementation

[ ] The inventory is created if the correct the items are added to the template.

Resources

https://docs.google.com/spreadsheets/d/1bBXY1PSHX6L4SkG1fco3eFLM5in7CeRqBFWThWQ5nK0/edit?usp=sharing

Wambere commented 2 weeks ago

@dubdabasoduba looks like the issue here is the url provided https://unicef-mg-eusm-preview.smartregister.org/fhir/... returns a 400:Bad request but https://gateway.unicef-mg-eusm-preview.smartregister.org/fhir/... (adding gateway. in the beginning) returns the expected results

can you try change the url in the config file?

Did the command work for other imports with this same url? Or was that change effected maybe while you were already in the middle of testing?

dubdabasoduba commented 2 weeks ago

@Wambere

I used the same server all through. It's the same one used by OpenSRP web

We usually deploy 2 instances of HAPI.

Has authentication which is https://unicef-mg-eusm-preview.smartregister.org
Has no auth and uses the gateway https://gateway.unicef-mg-eusm-preview.smartregister.org

both point to the same DB

Here are my results for the server in point 1

https://unicef-mg-eusm-preview.smartregister.org/fhir/List/_search?_count=1&subject=Location%2Fe01047f8-c160-4ff7-8a8d-d46eb1968f95

{
    "resourceType": "Bundle",
    "id": "331cf120-d519-41bd-ad61-45402cc3423b",
    "meta": {
        "lastUpdated": "2024-07-11T11:42:52.787+00:00"
    },
    "type": "searchset",
    "total": 0,
    "link": [
        {
            "relation": "self",
            "url": "https://unicef-mg-eusm-preview.smartregister.org/fhir/List/_search?_count=1&subject=Location%2Fe01047f8-c160-4ff7-8a8d-d46eb1968f95"
        }
    ]
}

Wambere commented 2 weeks ago

@dubdabasoduba that is the expected result. From the above error, what you were initially getting was not a JSON response, from the fact that it failed when trying to decode the response. My assumption was the change in the url, because that would cause a html 400 response, but it could have been anything else. But with the above response, the code should work as expected. You don't need the List to actually exists. 0 is a perfectly okay response. It just checks first so that it does not try to recreate one that already exists (so importing the same file multiple times won't cause issues)

dubdabasoduba commented 2 weeks ago

@Wambere I will try this again and revert

dubdabasoduba commented 6 days ago

@Wambere I retested this and it seems to work well. Closing the ticket

onaio / fhir-tooling