dfo-mar-odis / saraDataScraping

Repo to hold code and project management for the SARA data scraping project
MIT License
0 stars 0 forks source link

Design software components for pipeline -SARAdatascraping #15

Closed stoyelq closed 2 years ago

stoyelq commented 2 years ago

Proposed Change/Activity

Create a high level overview of the main class components for a potential data pipeline (writing outline of #17 ).

Why is this important

This will organize the code into discrete blocks which will help in keeping the codebase simple and modular.

Additional Context

The design will be updated as the project needs evolve

Actions

Expand and review the initial overview of all the potential classes and code components for this project to ensure that it is sufficient to handle all of the user stories.

Perceived Effort

3

Image

stoyelq commented 2 years ago

A question worth thinking about with regards to design is what kind of new features will need to be added as this project progresses. Do we expect to be adding more types of documents (e.g. pdf's generated with Rmd/Latex, different types of tables) or want to do more things with the existing ones (e.g. save to somewhere else, do data manipulation on the tables)?

Adding more types of documents works well with this design, add doing more things with them may be more conducive to a functional programming approach.

The model is not super involved yet, so I think the design should still be driven a bit by what the clients want( i.e. if they want a python tool, this OOP method should be okay) but this is worth having in mind at this stage.