search

nodestream-proj / nodestream

A Declarative framework for Building, Maintaining, and Analyzing Graph Data
https://nodestream-proj.github.io/docs/
Apache License 2.0
36 stars 10 forks source link

Add sqs queue extractor #318

Closed grantleehoffman closed 2 months ago

grantleehoffman commented 2 months ago

Adds Queue extractor and SQS connector for processing queue messages from a SQS queue. Has property delete_after_read which defaults to True. This will delete queue message batches after successfully reading to pipeline from extractor step.

Future Feature: Add in tracking of message processing through pipeline and delete once all writes are finished for each message.

codecov[bot] commented 2 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 97.74%. Comparing base (4b0b796) to head (df9eee5).

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #318 +/- ## ========================================== + Coverage 97.70% 97.74% +0.04% ========================================== Files 139 142 +3 Lines 4662 4754 +92 ========================================== + Hits 4555 4647 +92 Misses 107 107 ``` | [Flag](https://app.codecov.io/gh/nodestream-proj/nodestream/pull/318/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=nodestream-proj) | Coverage Δ | | |---|---|---| | [3.10-macos-latest](https://app.codecov.io/gh/nodestream-proj/nodestream/pull/318/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=nodestream-proj) | `97.74% <100.00%> (+0.04%)` | :arrow_up: | | [3.10-ubuntu-latest](https://app.codecov.io/gh/nodestream-proj/nodestream/pull/318/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=nodestream-proj) | `97.74% <100.00%> (+0.04%)` | :arrow_up: | | [3.10-windows-latest](https://app.codecov.io/gh/nodestream-proj/nodestream/pull/318/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=nodestream-proj) | `97.70% <100.00%> (+0.04%)` | :arrow_up: | | [3.11-macos-latest](https://app.codecov.io/gh/nodestream-proj/nodestream/pull/318/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=nodestream-proj) | `97.74% <100.00%> (+0.04%)` | :arrow_up: | | [3.11-ubuntu-latest](https://app.codecov.io/gh/nodestream-proj/nodestream/pull/318/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=nodestream-proj) | `97.74% <100.00%> (+0.04%)` | :arrow_up: | | [3.11-windows-latest](https://app.codecov.io/gh/nodestream-proj/nodestream/pull/318/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=nodestream-proj) | `97.70% <100.00%> (+0.04%)` | :arrow_up: | | [3.12-macos-latest](https://app.codecov.io/gh/nodestream-proj/nodestream/pull/318/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=nodestream-proj) | `97.74% <100.00%> (+0.04%)` | :arrow_up: | | [3.12-ubuntu-latest](https://app.codecov.io/gh/nodestream-proj/nodestream/pull/318/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=nodestream-proj) | `97.74% <100.00%> (+0.04%)` | :arrow_up: | | [3.12-windows-latest](https://app.codecov.io/gh/nodestream-proj/nodestream/pull/318/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=nodestream-proj) | `97.70% <100.00%> (+0.04%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=nodestream-proj#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

yasonk commented 2 months ago

@grantleehoffman Should this have an update in the documentation? Maybe worth adding it to the docs here: https://nodestream-proj.github.io/docs/docs/reference/extractors/

grantleehoffman commented 2 months ago

I will add an update to the public docs once these changes have gone through review and are finalized.