search

nodestream-proj / nodestream

A Declarative framework for Building, Maintaining, and Analyzing Graph Data
https://nodestream-proj.github.io/docs/
Apache License 2.0
37 stars 11 forks source link

Cache timestamps to speed up ingest #262

Closed zprobst closed 8 months ago

zprobst commented 8 months ago

Was playing around with some rust code over the weekend. On a hunch, I had a feeling that getting datetimes from pandas was going to be slow and alas... it was. This implements a time cache so that we don't get Pandas datetimes 1000s of times a second.

Current Performance

old

After Optimization

new
codecov[bot] commented 8 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (66d0953) 96.05% compared to head (317db90) 96.06%.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #262 +/- ## ======================================= Coverage 96.05% 96.06% ======================================= Files 133 133 Lines 3959 3968 +9 ======================================= + Hits 3803 3812 +9 Misses 156 156 ``` | [Flag](https://app.codecov.io/gh/nodestream-proj/nodestream/pull/262/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=nodestream-proj) | Coverage Δ | | |---|---|---| | [3.10-macos-latest](https://app.codecov.io/gh/nodestream-proj/nodestream/pull/262/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=nodestream-proj) | `?` | | | [3.10-ubuntu-latest](https://app.codecov.io/gh/nodestream-proj/nodestream/pull/262/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=nodestream-proj) | `96.06% <100.00%> (+<0.01%)` | :arrow_up: | | [3.10-windows-latest](https://app.codecov.io/gh/nodestream-proj/nodestream/pull/262/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=nodestream-proj) | `95.97% <100.00%> (+<0.01%)` | :arrow_up: | | [3.11-macos-latest](https://app.codecov.io/gh/nodestream-proj/nodestream/pull/262/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=nodestream-proj) | `96.06% <100.00%> (+<0.01%)` | :arrow_up: | | [3.11-ubuntu-latest](https://app.codecov.io/gh/nodestream-proj/nodestream/pull/262/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=nodestream-proj) | `?` | | | [3.11-windows-latest](https://app.codecov.io/gh/nodestream-proj/nodestream/pull/262/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=nodestream-proj) | `95.97% <100.00%> (+<0.01%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=nodestream-proj#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.