search

codeforIATI / iati-datastore

An open-source datastore for IATI data with RESTful web API providing XML, JSON, CSV plus ETL tools
https://datastore.codeforiati.org
Other
1 stars 1 forks source link

Problems with data import #354

Open odscjames opened 2 years ago

odscjames commented 2 years ago

Problems were seen with this. The worker had the following Python error repeated:


   File "/home/iatidatastoreclassic/iatidatastoreclassic/iati_datastore/iatilib/crawler.py", line 303, in update_dataset
     resource = fetch_resource(dataset, ignore_hashes)
   File "/home/iatidatastoreclassic/iatidatastoreclassic/iati_datastore/iatilib/crawler.py", line 117, in fetch_resource
     content = d.raw_xml
   File "/home/iatidatastoreclassic/iatidatastoreclassic/.ve/lib/python3.8/site-packages/iatikit/data/dataset.py", line 68, in raw_xml
     with open(self.data_path, 'rb') as handler:
   TypeError: expected str, bytes or os.PathLike object, not NoneType

353 should provide an immediate fix, but making this issue to make work visible and get to the bottom of this.

odscjames commented 2 years ago

One issue that came up is that the above error was not seen in Sentry. I'm not sure why; sentry is set up for the CLI (just verified). At first I thought it was RQ catching the error and putting it in it's own logs, so sentry doesn't get a chance to see it. Sentry does have RQ integrations that we may have needed - https://docs.sentry.io/platforms/python/guides/rq/ But I tried locally and an error in a task does get reported to sentry without that integration, so I'm not sure what is happening.

odscjames commented 2 years ago

Fixed an issue in server monitoring so we get alerts on unhealthy status in future.