Test/Review the "higher level of data fetching" with new sources

[arunge]

@isomemo the "higher level of data fetching" was added to the iso-app beta. The next step regarding this is to

• [ ] test the ETL with a new schema and new data source
• [ ] if necessary fix issues with the etl

Please feel free to provide the new sources.

Originally posted by @arunge in https://github.com/Pandora-IsoMemo/iso-app/issues/69#issuecomment-1593342068

Let me know, when you have collected new sources in form of

• url's,
• files, or
• databases (here we would have to think about a secure way to handle credentials dependent on your typical use cases) For testing purpose it would be easier to start with files or url's.

[arunge]

@isomemo as just discussed:

• you can find the description on how to add a new source in the ReadMe of iso-data
• please test the described process
• please let me know if something is not clear or if you observe any other issues

Thanks in advance!

[isomemo]

@arunge please assign Carlo to this. I cannot remember his GitHub name :-)

[arunge]

@CarloCocozza we implemented the option to select different networks (schemas) of data sources. However, currently there is only one network available on the database. That is, new sources must be integrated. We use an ETL (extract-transform-load) process to keep our data up-to-date, an overview can be found here: https://github.com/Pandora-IsoMemo/iso-data/blob/main/README.md

We would like to test the process of adding a new data source:

In order to add a new data source one needs to:

• have new data either retrieved from a local or remote file or from a sql database
• install and load the iso-data package locally (if you use RStudio, please check https://happygitwithr.com/rstudio-git-github)
• and go through the steps from the ReadMe

We developed helper functions (createNewDBSource() and createNewFileSource()) that create required templates in order to add a new source to the existing ETL. Please read the section add-a-new-data-source carefully to understand how to set parameters of createNewDBSource() or createNewFileSource().

If one has created a new script for a new source, the changes must be commited to github on a separate feature branch. One has to open a pull request to propose the changes. We will check (review and test) the changes in a next step. If all works as expected, we can add the new source to our ETL.

Please let me know if you have questions regarding the ReadMe or if you observe any errors or warnings. If you have issues with read/write access regarding github, please also let me know.

[arunge]

@CarloCocozza I added you as a collaborator to this repository. Currently, I cannot assign you to this issue, I believe you must first accept the invitation.