search

NOAA-ORR-ERD / LibGOODS

Library for accessing data useful for the NOAA / GNOME model
https://libgoods.readthedocs.io/en/latest/
Other
1 stars 2 forks source link

model_catalogs needs more fault tolerance #24

Open jay-hennen opened 1 year ago

jay-hennen commented 1 year ago

Specifically, when running the complete_source_catalog function.

Here are my thoughts:

  1. Failures should simply be omitted from the resulting catalog. It's easier to work with a catalog where we can assume all members are complete and successful.
  2. It may work to log the failures along with the successes in a 'access report' json log file that can also be a good place to put other meta-information about each model's retrieval. E.G Network time, download size, timestamp, etc.
  3. Failures should produce console log messages at the time of their failure as well of course.
ChrisBarker-NOAA commented 1 year ago

Agreed, with one suggested modification:

log the failures along with the successes in a 'access report' json log file

we may want to go with "ordinary" logging.

Though a I think that kind of meta-info should be saved somewhere -- maybe in the same catalog json as everything else?

NOTE: if we do set up a logger, "loguru" is pretty nice -- easier to configure that the built in logging package:

https://pypi.org/project/loguru/

lukecampbell commented 1 year ago

@kthyng I think this is fixed?