Closed hmtbr closed 1 month ago
Can this be scaled up by implementing it as a transform.
No. The plan is to release this as an open source package from which we can create additional transform. @daw3rd please remove your request for change. Thanks
@daw3rd Thank you for reviewing this. I added overview.md
to describe example usage. Could you take a look at the documentation? Let me know if I need additional updates.
In order to make the repo feel a bit more cohesive, should we rename this top-level directory to data-connector-lib to align with data-processing-lib?
Renamed the top level folder to data-connector-lib
.
Thank you @daw3rd !
Why are these changes needed?
We want to cover the full life cycle for data acquisition and data processing in DPK. This PR will add the core library to perform data acquisition from websites.
Related issue number (if any).
633