search

hasadna / knesset-data-pipelines

Main repository for Open Knesset project - contains the knesset data scrapers and processing pipelines
https://oknesset.org/
MIT License
14 stars 26 forks source link

Outdated README: Misleading Information About Pipelines and Airflow #229

Open yanirmr opened 1 year ago

yanirmr commented 1 year ago

Hello! 😅 I must admit, I've had quite an adventure trying to understand and work with the pipelines. After spending a few hours trying to get the system up and running and debugging issues, I realized that the main README for the project focuses heavily on datapackages, while in reality, the current implementation is based on Airflow.

As a newcomer to the project, this crucial information escaped me, and I ended up spending valuable time attempting to run things on the wrong infrastructure. Although it might be due to my inexperience with the project, I believe that the README could benefit from an update to provide clearer guidance for users like myself.

I kindly request that the README be updated to accurately reflect the current status of the project, with a clear emphasis on the use of Airflow for managing pipelines. This would help newcomers get up to speed more quickly and avoid potential confusion.

Thank you for understanding, and I appreciate your attention to this matter! 😊

OriHoch commented 1 year ago

yes, we really should improve the README, but the datapackage pipelines are still relevant. Until we do that I suggest that you contact me on slack and we schedule a meeting to go over it and help you get started with the project.