Summary:
This pull request introduces updates to enhance logging, authentication, and error handling in api/src/scripts/load_dataset_on_create.py and related files.
Replaced standard logging with a custom logger in load_dataset_on_create.py for more structured, consistent log messages.
Adjusted the logging level for downstream task triggers in populate_db_gtfs.py to ensure critical events are logged appropriately.
Ensured the message publishing process in load_dataset_on_create.py now blocks until confirmation is received, preventing potential issues with unacknowledged messages.
Issue Explanation:
Several issues were identified and addressed in this update:
Lack of Confirmation for Message Publishing: Previously, we did not wait for confirmation of message publication to Pub/Sub topic, which led to silent failures. This issue went unnoticed after moving the database instance from DEV to QA, as logs did not indicate the failure caused by a permission error.
Misconfigured Project ID and Pub/Sub Topic: After fixing the permission issue, I discovered that the project_id and pubsub_topic_name were incorrectly set. These variables were initialized at a high level in the script before the environment file was fully loaded, resulting in attempts to publish to the default (dev) topic instead of the QA topic or PROD topic.
Testing Instructions:
To test the updates:
Recently added feeds (e.g., stable_id > mdb-2155) should all have a corresponding dataset in QA and PROD environments.
For example, you can check mdb-2156 in QA here: QA Feed Example.
Please make sure these boxes are checked before submitting your pull request - thanks!
[x] Run the unit tests with ./scripts/api-tests.sh to make sure you didn't break anything
[ ] Add or update any needed documentation to the repo
Summary: This pull request introduces updates to enhance logging, authentication, and error handling in
api/src/scripts/load_dataset_on_create.pyand related files.load_dataset_on_create.pyfor more structured, consistent log messages.populate_db_gtfs.pyto ensure critical events are logged appropriately.load_dataset_on_create.pynow blocks until confirmation is received, preventing potential issues with unacknowledged messages.Issue Explanation:
Several issues were identified and addressed in this update:
project_idandpubsub_topic_namewere incorrectly set. These variables were initialized at a high level in the script before the environment file was fully loaded, resulting in attempts to publish to the default (dev) topic instead of the QA topic or PROD topic.Testing Instructions:
To test the updates:
stable_id>mdb-2155) should all have a corresponding dataset in QA and PROD environments.mdb-2156in QA here: QA Feed Example.Please make sure these boxes are checked before submitting your pull request - thanks!
./scripts/api-tests.shto make sure you didn't break anything