[Lake][ETL] ETL.md - Explain architecture & ux end-to-end - Githubissues

oceanprotocol / pdr-backend

Instructions & code to run predictoors, traders, more.

Apache License 2.0

22 stars 15 forks source link

[Lake][ETL] ETL.md - Explain architecture & ux end-to-end #1002

Open idiom-bytes opened 1 month ago

idiom-bytes commented 1 month ago

Background / motivation

To integrate duckDB (#685)[https://github.com/oceanprotocol/pdr-backend/issues/685] the user should be able to follow the README.md, get their lake setup, and complete all the work end-to-end without any issues.

Further, if the user should experience any issues, they should be able to resolve/heal the process effortlessly.

DoD

Write README.md for updated lake, and verify that it's working end-to-end

Tasks:

[ ] Write new lake README.md
[ ] Test README.md end-to-end
[ ] Documentation is working as expected

KatunaNorbert commented 1 month ago

Reviewed the README and have some feedback:

[ ] Add information about where the data comes from and what kind of data we are ingesting into the lake
[ ] There is no information about the data available, tables etc. I would expect to visually see the data that I'm going to get before going trough the lake setup process
[ ] Add a simple example containing the command the users has to execute to go trough the etl end to end
[ ] The README is long and we don't have any legend on the top to make it easy to see the content and navigate trough different sections. Add a legend on the top of the page, example: