ANIALLATOR114 / SimplyTransport

SimplyTransport - API - Website - Ingesting and presenting Transport Information
https://simplytransport.ie
Apache License 2.0
2 stars 3 forks source link

Add unique constraint to realtime updates, sync profiler #144

Closed ANIALLATOR114 closed 3 days ago

ANIALLATOR114 commented 3 days ago

Summary by Sourcery

Add a unique constraint to the 'rt_stop_time' table to ensure data integrity and modify the import process to use upsert operations. Introduce a new profiling decorator for synchronous functions to facilitate performance analysis. Update CLI documentation for clarity.

New Features:

Enhancements:

Build:

Documentation:

sourcery-ai[bot] commented 3 days ago

Reviewer's Guide by Sourcery

This pull request implements a unique constraint for real-time updates and adds a synchronous profiler. The main changes include modifying the GTFS real-time importer to use upsert operations, adding a unique constraint to the RTStopTimeModel, and introducing a new synchronous profiling decorator.

Sequence Diagram

sequenceDiagram
    participant C as Client
    participant I as Importer
    participant DB as Database
    C->>I: import_stop_times(data)
    I->>I: Process stop times
    I->>DB: Prepare upsert statement
    I->>DB: Execute upsert
    DB-->>I: Confirmation
    I-->>C: Import complete

File-Level Changes

Change Details Files
Implement upsert operation for real-time stop times
  • Replace direct ORM object creation with dictionary-based approach
  • Implement SQLAlchemy's insert().on_conflict_do_update() for upsert operation
  • Update error handling to use the new upsert approach
SimplyTransport/lib/gtfs_realtime_importers.py
Add unique constraint to RTStopTimeModel
  • Add UniqueConstraint for stop_id, trip_id, stop_sequence, and dataset
  • Create a new Alembic migration to add the unique constraint to the database
SimplyTransport/domain/realtime/stop_time/model.py
migrations/main/versions/2024_09_30-5889d747e8a0_add_unique_constraint.py
Introduce synchronous profiling decorator
  • Add profile_sync decorator function for synchronous function profiling
  • Implement cProfile and pstats for performance analysis
SimplyTransport/lib/profiling.py
Minor documentation update
  • Update docstring for importrealtimevehicles function to specify it's for vehicle data
SimplyTransport/cli.py

Tips and commands #### Interacting with Sourcery - **Trigger a new review:** Comment `@sourcery-ai review` on the pull request. - **Continue discussions:** Reply directly to Sourcery's review comments. - **Generate a GitHub issue from a review comment:** Ask Sourcery to create an issue from a review comment by replying to it. #### Customizing Your Experience Access your [dashboard](https://app.sourcery.ai) to: - Enable or disable review features such as the Sourcery-generated pull request summary, the reviewer's guide, and others. - Change the review language. - Add, remove or edit custom review instructions. - Adjust other review settings. #### Getting Help - [Contact our support team](mailto:support@sourcery.ai) for questions or feedback. - Visit our [documentation](https://docs.sourcery.ai) for detailed guides and information. - Keep in touch with the Sourcery team by following us on [X/Twitter](https://x.com/SourceryAI), [LinkedIn](https://www.linkedin.com/company/sourcery-ai/) or [GitHub](https://github.com/sourcery-ai).