search

NHMDenmark / DaSSCo-asset-service

DaSSCo asset service is part of DaSSCo storage system
0 stars 0 forks source link

Structures for files on slurm, integration, ingestion and ndrive #20

Open Baeist opened 6 months ago

Baeist commented 6 months ago

Integration: 

  - app_home_dir
    - Files
      - NewFiles
        - {pipeline_name}*
          - {batch_date}*
            - {guid}*
              - {guid}.tif,  {guid}.json, {guid}_jobs.json etc
      - InProcess
        - {pipeline_name}*
          - {batch_date}*
            - {guid}*
              - {guid}.tif,  {guid}.json, {guid}_jobs.json etc 
      - UpdatedFiles
        - {pipeline_name}*
          - {batch_date}*
            - {guid}*
              - {guid}.tif,  {guid}.json, {guid}_jobs.json etc
      - Error
        - {pipeline_name}*
          - {batch_date}*
            - {guid}*
              - {guid}.tif,  {guid}.json, {guid}_jobs.json etc

Ndrive: 

  - upload_dir
      - {workstation_name}*
        - {batch_date}*
            - {guid}.tif,  {guid}.json

Slurm: 

  - our_home_dir
    - imported_assets
      - {workstation_name}*
        - {batch_date}*
         - {batch_date}_list.json, {guid}*
            - {guid}.tif,  {guid}.json, {guid}_jobs.json {guid}_httplink.json

There can be multiple unique directories when noted with *