search

USF-IMARS / imars_dags

:leaves: USF IMaRS Airflow DAGs
0 stars 0 forks source link

s3 processing DAG #68

Closed 7yl4r closed 6 years ago

7yl4r commented 6 years ago

from @dotis:

Here are the files I am using to process S3 files using l2gen on userproc. Right now, the filenames and paths are hard coded in there. If you have any ideas about how best to carry this out and where to put files with minimal effort, please let me know.

files in this email

the .sh:

#!/bin/bash

# set environment variabless
export OCSSWROOT=/opt/ocssw
source $OCSSWROOT/OCSSW_bash.env

echo running l2gen...
l2gen ifile=/home1/scratch/epa/satellite/modis/S3_USVI/S3_raw/S3A_OL_1_EFR____20170118T142844_20170118T143144_20180328T191913_0179_013_210_2700_LR2_R_NT_002.SEN3/xfdumanifest.xml ofile=/home1/scratch/epa/satellite/modis/S3_USVI/S3_proc/S3A_EFR_20170118_2.L2 par=/home1/scratch/epa/satellite/modis/S3_USVI/IMaRS_S3_OLCI_l2gen.par

echo running mapping graph...
/opt/snap_6_0/bin/gpt /home1/scratch/epa/satellite/modis/S3_USVI/map_USVI_S3_OLCI.xml -t /home1/scratch/epa/satellite/modis/S3_USVI/S3_map/20170118.map -f GeoTIFF /home1/scratch/epa/satellite/modis/S3_USVI/S3_proc/S3A_EFR_20170118_2.L2
7yl4r commented 6 years ago

s3 filename spec

dotis commented 6 years ago

I wrote up a new .sh file. If filename is defined like so,

FILENAME=$1 # eg S3A_OL_1_EFR____20170118T

I can't make the following line work. It doesn't seem to like the *.SEN3 and can't open the file. I must be getting the syntax wrong.

l2gen ifile=$DATA_DIR/S3_raw/$FILENAME*.SEN3/xfdumanifest.xml ofile=$DATA_DIR/S3_proc/$FILENAME.L2 par=$DATA_DIR/IMaRS_S3_OLCI_l2gen.par

I think the best way is just to shorten the filename of the *.SEN3 folder manually.

On Sun, May 27, 2018 at 8:49 PM, Tylar notifications@github.com wrote:

s3 filename spec https://sentinel.esa.int/web/sentinel/user-guides/sentinel-3-slstr/naming-convention

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/USF-IMARS/imars_dags/issues/68#issuecomment-392392852, or mute the thread https://github.com/notifications/unsubscribe-auth/AZUtjdY1fugqqhjEJQQRV8cyUEZ7BuHNks5t20m1gaJpZM4UPf2f .

--

Dan Otis, PhDScientific ResearcherUniversity of South FloridaCollege of Marine ScienceInstitute for Marine Remote Sensing (IMaRS)140 7th Ave SouthSt. Petersburg, FL 33701