meltano / sdk

Write 70% less code by using the SDK to build custom extractors and loaders that adhere to the Singer standard: https://sdk.meltano.com
https://sdk.meltano.com
Apache License 2.0
94 stars 68 forks source link

refactor: Added a generic `FileStream` (still in active development!) #2654

Open edgarrmondragon opened 3 weeks ago

edgarrmondragon commented 3 weeks ago

References:

codspeed-hq[bot] commented 3 weeks ago

CodSpeed Performance Report

Merging #2654 will not alter performance

Comparing 2648-feat-add-a-generic-filestream-interface (4628468) with main (e997deb)

Summary

✅ 6 untouched benchmarks

codecov[bot] commented 3 weeks ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Please upload report for BASE (main@f1e8114). Learn more about missing BASE report.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #2654 +/- ## ======================================= Coverage ? 90.46% ======================================= Files ? 62 Lines ? 4992 Branches ? 974 ======================================= Hits ? 4516 Misses ? 330 Partials ? 146 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

edgarrmondragon commented 2 days ago

These are working now

FTP

{
    "filesystem": "ftp",
    "path": "fixtures/csv",
    "read_mode": "one_stream_per_file",
    "delimiter": "\t",
    "ftp": {
        "host": "127.0.0.1",
        "port": 21,
        "username": "my_ftp_user",
        "password": "my_ftp_password"
    }
}

SFTP

{
    "filesystem": "sftp",
    "path": "fixtures/csv",
    "read_mode": "one_stream_per_file",
    "delimiter": "\t",
    "sftp": {
        "host": "127.0.0.1",
        "port": 2022,
        "username": "my_ftp_user",
        "password": "my_ftp_password"
    }
}