ML4GW / aframe

Detecting binary black hole mergers in LIGO with neural networks
MIT License
6 stars 16 forks source link

Add luigi `PathParameter` for dynamically parsing s3 or local strings into paths #162

Closed EthanMarx closed 5 months ago

EthanMarx commented 5 months ago
  1. Adds a luigi PathParameter object that will dynamically create either a cloudpath.S3Path or pathlib.Path object depending on if the strings starts with s3://. s3 doesn't have the concept of "directories" so paths like s3://aframe/data/ is treated as a different location than s3://aframe/data which can lead to undesired behavior or user error. Using the cloudpath.S3Path object will mitigate this problem, treating the paths equivalently.

  2. Within our aframe.tasks module, we can now get remove os.path.join in favor of the pathlib.Path / notation, which I find much cleaner. We can also take advantage of pathlib.Path built in methods.

  3. Refactors out the load_prior method into a unified spot for the Waveform and TimeslideWaveform tasks to use.

  4. Consolidates Tune and Sandbox pipelines that really only differ in the training task that is used