Closed MinuraPunchihewa closed 1 year ago
Hey @MinuraPunchihewa thanks!👌 from what I understand from you screenshots, if I specify outputs_paths then the pipeline exécutions folder is not created anymore and go straight to the location specified. Is that correct ?
Hey @jravenel, Yes, that is correct.
Ok. I think @FlorentLvr would argue it will be best to keep the name of the folder pipeline_executions every time. Otherwise things can get messy, or give us the job to create a new folder, name it, etc when we don't really need that effort.
Oh, I don't quite understand. I worked on this to resolve this issue, https://github.com/jupyter-naas/naas/issues/373
Ok. I think @FlorentLvr would argue it will be best to keep the name of the folder pipeline_executions every time. Otherwise things can get messy, or give us the job to create a new folder, name it, etc when we don't really need that effort.
I don't understand what you mean @jravenel! This PR seems to resolve the issue i had 🙈
@MinuraPunchihewa, Let's summarize to ensure everything functions as intended:
The question I was raising is: Do we want to have: A/ all the pipeline executions in any folder (it will create a lot of folders in the directory) B/ specify a folder (outputs) and keep a pipeline_executions folder in the target folder generated automatically
@MinuraPunchihewa @FlorentLvr
@MinuraPunchihewa, Let's summarize to ensure everything functions as intended:
- If no value is specified for "output_dir," the pipeline executions will be stored in a folder named "pipeline_executions," located at the position where the pipeline notebook is invoked. This maintains the same behavior as before.
- If a specific path is provided for "output_dir," such as "/home/ftp/data-product/outputs/pipeline_executions," all executions will be stored in that specified location when the pipeline is executed. Is my understanding accurate?
@FlorentLvr Yes, you are right.
And my point is why do I need to specify /pipeline_executions in the output dir? Can't we have pipeline_executions created automatically ?
And my point is why do I need to specify /pipeline_executions in the output dir? Can't we have pipeline_executions created automatically ?
@jravenel, you don't. It is default parameter in the function. @MinuraPunchihewa
@MinuraPunchihewa, Let's summarize to ensure everything functions as intended:
- If no value is specified for "output_dir," the pipeline executions will be stored in a folder named "pipeline_executions," located at the position where the pipeline notebook is invoked. This maintains the same behavior as before.
- If a specific path is provided for "output_dir," such as "/home/ftp/data-product/outputs/pipeline_executions," all executions will be stored in that specified location when the pipeline is executed. Is my understanding accurate?
@FlorentLvr Yes, you are right.
@MinuraPunchihewa, sounds good to me! You can merge the PR once all checks are valid :)
Hey @Dr0p42, @FlorentLvr, Do you know why the checks are failing?
Hey @Dr0p42, @FlorentLvr, Do you know why the checks are failing?
@MinuraPunchihewa it's coming from the linter, go to GitHub Action you'll be able to have infos.
Hey @jravenel, Yes, I saw that, but it's not clear to me what the issue is.
Calling Mr @Dr0p42 for help here 🙏
Kudos, SonarCloud Quality Gate passed!
0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells
No Coverage information
0.0% Duplication
Ok @MinuraPunchihewa @jravenel this is finally fixed :)
;) thanks @Dr0p42. We will be implementing it today or tomorrow.
This Pull Request aims to allow users to define a custom path to store the results of their Pipeline executions. This is done by using the
outputs_path
parameter in therun()
method of thePipeline
class.Given below is an example of how this works,
As shown here, the![image](https://github.com/jupyter-naas/naas/assets/49385643/01362760-69ed-44ce-b5c9-f77fc237c556)
outputs
directory is created as specified above and the results of the pipeline executions are stored in separate sub-directories within it,The contents of the![image](https://github.com/jupyter-naas/naas/assets/49385643/9119fce7-876b-440c-8cb5-28ed91163994)
outputs
directory,If a path is not specified, the results will be stored within the
pipeline_executions
directory, by default.Note: This directory is also visible in the above screenshot, because I have the run the Pipeline twice; once without specifying the
outputs_path
and a second time by passing the argument.