MobilityData / mobility-feed-api

Apache License 2.0
9 stars 3 forks source link

feat: add support for storing and validating GBFS files in GCP storage #695

Closed cka-y closed 2 months ago

cka-y commented 3 months ago

Summary:

Closes #566 by adding support for storing GBFS files from an auto-discovery URL in a Cloud Storage bucket. The updates include the creation of a new gbfs.json file with paths pointing to the stored files in the bucket, validating the feeds, and storing the necessary information in the database. The changes involve the creation of the gbfs-validator-batch and gbfs-validator-pubsub functions to support this new functionality, ensuring that GBFS feeds are properly validated and stored.

Expected behavior:

This pull request introduces the following behavior:

Testing tips:

  1. Trigger the batch function:

    • Execute the following command to trigger the batch function:
      curl -m 550 -X POST https://northamerica-northeast1-mobility-feeds-dev.cloudfunctions.net/gbfs-validator-batch \
      -H "Authorization: bearer $(gcloud auth print-identity-token)" \
      -H "Content-Type: application/json"
    • Alternatively, trigger the Cloud Scheduler by selecting gbfs-validator-batch-scheduler-dev and clicking Force Run.
  2. The gbfs-validator-pubsub function should be triggered automatically. You can verify the logs to ensure everything is functioning correctly: Pub/Sub Function Logs.

  3. Check the stored feeds in the specified Cloud Storage bucket. Ensure the gbfs.json file and the validation report are correctly generated: GBFS Storage Bucket.

  4. Verify that the database content in the DEV env reflects the expected changes and updates.

Testers are invited to follow the tips AND to try anything they deem relevant outside the bounds of the testing tips.

Please make sure these boxes are checked before submitting your pull request - thanks!

cka-y commented 3 months ago

The integration tests are expected to fail because of the database changes