Closed keegansmith21 closed 1 year ago
:exclamation: No coverage uploaded for pull request base (
develop@356b14f
). Click here to learn what that means. Patch coverage: 98.79% of modified lines in pull request are covered.
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.
Originally I had attempted to alter the ONIX parser execution to use a BashOperator, rather than pipe the command to a subprocss in python. This proved to be unexpectedly difficult for many reasons. This may be an easier task once we upgrade airflow and have access to dynamic tasks. For now, I have updated the parser call to run without the shell environment (shell=False).
I have consolidated the onix-related functions into a new file called onix.py. The reason for this is that there are three telescopes that use common onix related functions and I would prefer not to have to import a function from a telescope in another telescope.
Due to the recent refactor, the sftp file directories have changed. This necessitates a change to both the sftp server directories and also the sftp root directory in the onix telescope. With the consideration that the SftpFolders class now generates a unique folder based on the dag_id, I think that it is unnecessarily complex to have an alternate sftp root directory for each onix telescope. Therefore, I have set the default to the root of the filesystem ("/"). Should we require more functionality, this will need to be changed, but it is not in the foreseeable future.