Adds a luigiPathParameter object that will dynamically create either a cloudpath.S3Path or pathlib.Path object depending on if the strings starts with s3://. s3 doesn't have the concept of "directories" so paths like s3://aframe/data/ is treated as a different location than s3://aframe/data which can lead to undesired behavior or user error. Using the cloudpath.S3Path object will mitigate this problem, treating the paths equivalently.
Within our aframe.tasks module, we can now get remove os.path.join in favor of the pathlib.Path/ notation, which I find much cleaner. We can also take advantage of pathlib.Path built in methods.
Refactors out the load_prior method into a unified spot for the Waveform and TimeslideWaveform tasks to use.
Consolidates Tune and Sandbox pipelines that really only differ in the training task that is used
Adds a
luigiPathParameterobject that will dynamically create either acloudpath.S3Pathorpathlib.Pathobject depending on if the strings starts withs3://.s3doesn't have the concept of "directories" so paths likes3://aframe/data/is treated as a different location thans3://aframe/datawhich can lead to undesired behavior or user error. Using thecloudpath.S3Pathobject will mitigate this problem, treating the paths equivalently.Within our
aframe.tasksmodule, we can now get removeos.path.joinin favor of thepathlib.Path/notation, which I find much cleaner. We can also take advantage ofpathlib.Pathbuilt in methods.Refactors out the
load_priormethod into a unified spot for theWaveformandTimeslideWaveformtasks to use.Consolidates
TuneandSandboxpipelines that really only differ in the training task that is used