Accelerate aggregation and parse

takohei commented 4 months ago

Close Issues

1
5
MIERUNE/GTFS-GO#77

Benchmark

data set

name	source	size	file
Takushoku Bus	the fixture of test	small	fx_TakushokuBus.zip
Tokyo Toei Bus	#6	medium	P6_ToeiBus.zip
Vancouver TransLink	MIERUNE/GTFS-GO#77	large	G77_TransLink.zip
Swiss	#1	extra large	GTFS_FP2021_2021-12-08_09-10.zip

results

read_stops(): 1.2 - 1.7 times faster
read_routes() noshape: 8.0 - 12.7 times faster
read_routes() shape: 4.1 - 6.6 times faster
aggregator nounify: 2.9 - 3.9 times faster
aggregator unify: 21.0 - 37.2 times faster

remarks

Current read_routes()@parse.py has 2 bugs. I fix them temporary and use for measurement. Like this:

        # Point -> LineString: group by route_id and trip_id
        lines = merged.groupby(["route_id", "trip_id"])["stop_pt"].apply(tuple)
        line_df = lines.reset_index()[["route_id", "stop_pt"]].drop_duplicates()

Description（変更内容）

Accelerate and refactor for aggregation, especially for stop unification

Change the unification from apply per stops to process for whole dataframe.
Change merge procedures so that they are done on smaller dataframes.
Decrease max_distance_degree 0.01 to 0.003 and add process of join near groups.
- This change reduces the number of unified stops in test cases because there are stops with a same name that are 400 meters apart from each other.
Exclude parent stops from aggregation with no_unify_stops option.
- This change reduces the number of aggregated stops in the test case with no_unify_stops option.
Break large __aggregate_similar_stops() into small static methods.
Remove path_id and related procedures in read_route_frequency()

Add read_stop_relations() to Aggregator

Add read_stop_relations() that returns relations of stops and similar stops as a list of dictionaries.
Eliminate modification of gtfs data frames in the Aggregator class.
Eliminate GTFS-GO (gtfs_go_dialog.py) getting stop relations from inside Aggregator.

Acclerate and refactor read_routes()

Unify stop patterns by stop_id, not by line string.
Minimize number of coordinate pairing and linestring generation.
Fix sort procedure for stop sequence.
- The route collapse problem in PR #9 was caused by a difference in sorting behavior between Pandas 2.2 and 1.1 (built into QGIS 3.28).
Change the method of generating features from iterrows() to to_dict().
Break large read_routes() into some small methods.

Acclerate read_stops()

Change merge/join procedure to be faster.
Change the method of generating features from iterrows() to to_dict().

Add test cases for aggregation

Add test cases of delimiter, time filter.
Add tests for record values.
Set target date to feed_start_date.
Arrange the test cases from per-method to per-option.

Manual Testing（手動テスト）

Please check in QGIS on your Mac OS.

coderabbitai[bot] commented 4 months ago

ウォークスルー

GTFSデータ解析のためのgtfs_parserライブラリが大幅に更新されました。主な変更点はAggregatorクラスのメソッドの追加と削除、データフレームの処理方法の改善、およびGeoJSON機能生成のための新しい関数の導入です。これにより、より効率的かつ柔軟なデータ処理が可能になります。

変更

ファイル	変更概要
`gtfs_parser/aggregate.py`	`Aggregator`クラスのメソッドの追加・削除、`__init__`メソッドの更新
`gtfs_parser/gtfs.py`	`load_df`関数に`location_type`列の処理を追加
`gtfs_parser/parse.py`	停留所とルートの読み取り処理の変更、GeoJSON機能生成のための新しい関数
`tests/test_aggregator.py`	新しいテスト関数の追加

🐰✨
新しい道へと進むたび
コードの森を駆け巡る
データの流れを整え
地図上に軌跡を描く
変化を楽しむ我らが旅
🌟🌲🌲🌲🌲🌲🌲

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

- [X](https://twitter.com/intent/tweet?text=I%20just%20used%20%40coderabbitai%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20the%20proprietary%20code.%20Check%20it%20out%3A&url=https%3A//coderabbit.ai) - [Mastodon](https://mastodon.social/share?text=I%20just%20used%20%40coderabbitai%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20the%20proprietary%20code.%20Check%20it%20out%3A%20https%3A%2F%2Fcoderabbit.ai) - [Reddit](https://www.reddit.com/submit?title=Great%20tool%20for%20code%20review%20-%20CodeRabbit&text=I%20just%20used%20CodeRabbit%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20proprietary%20code.%20Check%20it%20out%3A%20https%3A//coderabbit.ai) - [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcoderabbit.ai&mini=true&title=Great%20tool%20for%20code%20review%20-%20CodeRabbit&summary=I%20just%20used%20CodeRabbit%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20proprietary%20code)

Tips

### Chat There are 3 ways to chat with [CodeRabbit](https://coderabbit.ai): - Review comments: Directly reply to a review comment made by CodeRabbit. Example: - `I pushed a fix in commit .` - `Generate unit testing code for this file.` - `Open a follow-up GitHub issue for this discussion.` - Files and specific lines of code (under the "Files changed" tab): Tag `@coderabbitai` in a new review comment at the desired location with your query. Examples: - `@coderabbitai generate unit testing code for this file.` - `@coderabbitai modularize this function.` - PR comments: Tag `@coderabbitai` in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples: - `@coderabbitai generate interesting stats about this repository and render them as a table.` - `@coderabbitai show all the console.log statements in this repository.` - `@coderabbitai read src/utils.ts and generate unit testing code.` - `@coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.` Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. ### CodeRabbit Commands (invoked as PR comments) - `@coderabbitai pause` to pause the reviews on a PR. - `@coderabbitai resume` to resume the paused reviews. - `@coderabbitai review` to trigger a review. This is useful when automatic reviews are disabled for the repository. - `@coderabbitai resolve` resolve all the CodeRabbit review comments. - `@coderabbitai help` to get help. Additionally, you can add `@coderabbitai ignore` anywhere in the PR description to prevent this PR from being reviewed. ### CodeRabbit Configration File (`.coderabbit.yaml`) - You can programmatically configure CodeRabbit by adding a `.coderabbit.yaml` file to the root of your repository. - Please see the [configuration documentation](https://docs.coderabbit.ai/guides/configure-coderabbit) for more information. - If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: `# yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json` ### Documentation and Community - Visit our [Documentation](https://coderabbit.ai/docs) for detailed information on how to use CodeRabbit. - Join our [Discord Community](https://discord.com/invite/GsXnASn26c) to get help, request features, and share feedback. - Follow us on [X/Twitter](https://twitter.com/coderabbitai) for updates and announcements.

codecov[bot] commented 4 months ago

Codecov Report

Attention: Patch coverage is 91.10169% with 21 lines in your changes are missing coverage. Please review.

Project coverage is 91.24%. Comparing base (98f7fad) to head (34daec8). Report is 1 commits behind head on main.

Files	Patch %	Lines
tests/test_aggregator.py	85.91%	10 Missing :warning:
gtfs_parser/aggregate.py	93.16%	8 Missing :warning:
gtfs_parser/parse.py	93.47%	3 Missing :warning:

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #11 +/- ## =========================================== + Coverage 80.61% 91.24% +10.63% =========================================== Files 8 8 Lines 294 377 +83 =========================================== + Hits 237 344 +107 + Misses 57 33 -24 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Kanahiro commented 3 months ago

LGTM!

MIERUNE / gtfs-parser