search

NHMDenmark / DaSSCo-Integration

This Repo will include integration of dassco storage from northtec
0 stars 0 forks source link

Structures for Slurm, Ndrive and IntegrationServer #19

Open Baeist opened 7 months ago

Baeist commented 7 months ago

Integration: 

  - app_home_dir
    - Files
      - NewFiles
        - {pipeline_name}*
          - {batch_date}*
            - {guid}*
              - {guid}.tif,  {guid}.json, {guid}_jobs.json etc
      - InProcess
        - {pipeline_name}*
          - {batch_date}*
            - {guid}*
              - {guid}.tif,  {guid}.json, {guid}_jobs.json etc 
      - Error
        - {pipeline_name}*
          - {batch_date}*
            - {guid}*
              - {guid}.tif,  {guid}.json, {guid}_jobs.json etc

Ndrive: 

  - upload_dir
      - {workstation_name}*
        - {batch_date}*
            - {guid}.tif,  {guid}.json

Slurm: 

  - our_home_dir
    - received
      - {batch_id}*
         - {batch_id}_list.json, {guid}*
            - {guid}.json

There can be multiple unique directories when noted with *

Baeist commented 3 months ago

Integration: 

  - app_home_dir
    - Files
      - NewFiles
        - {pipeline_name}*
          - {batch_date}*
            - {guid}*
              - {guid}.tif,  {guid}.json,  etc
      - InProcess
        - {pipeline_name}*
          - {batch_date}*
            - {guid}*
              - {guid}.tif,  {guid}.json, etc 
      - Error
        - {pipeline_name}*
          - {batch_date}*
            - {guid}*
              - {guid}.tif,  {guid}.json,  etc

Ndrive: 

  - upload_dir
      - {workstation_name}*
        - {batch_date}*
            - {guid}.tif,  {guid}.json

Slurm: 

  - our_home_dir
     - pipeline_dir
        - workingDirectory  
             - {batch_id}*
                - {guid}.json, {guid}.tif

There can be multiple unique directories when noted with *

Removed the _jobs.json. Updated slurm structure to reflect current testing environment.