+ automated sentinel ingest dag

[7yl4r]

Sentinel images from OBDAAC are not named by their coverage region, so we need to look at their coverage in order to download just the ROI we want.

We could do this using CMR like we did with the modis coverage checks.

[7yl4r]

It should absolutely be possible to download the S3 L1 for GoM with the my airflow script but as far as I can tell they are inexplicably missing from NASA's CMR.

The short_names I found in LAADS (search s3 here) are not returning anything (example CMR request)

and the collection-concept-ids in GCMD (example) are saying the "Concept-id is not valid.". (example CMR request)

I think we may need to report this to NASA and wait for a response before I can automate the ingest.

[7yl4r]

Aha. Found what we want on earthdata search. collection id is C1370679936-OB_DAAC. Looks like latest data there lags current date by ~15-days.

Also found this nice little form for generating an XML CMR query.

[7yl4r]

The granule timedelta seems to drift a few milliseconds and doesn't fall on a precise minute like modis products do. It looks like the granule timedelta is ~3min according to these LADS example files (first date in filepath is sensing start date).

[7yl4r]

@dotis : so, I set up an automated ingest of the s3 data from LADS, but there is a big delay between sensor time and when it is available in the CMR. The delay seems to vary from 15 days to over 30 days.

Is the automated ingest still helpful if the files coming in are 30 days old?

[dotis]

Maybe for some things, but not for what I need next week. I guess you'll have to go to the visual browser for now. ESA should have an API hub for S3 OLCI data soon.

https://scihub.copernicus.eu/s3/#/home

On Fri, Jun 1, 2018 at 1:20 PM, Tylar notifications@github.com wrote:

@dotis https://github.com/dotis : so, I set up an automated ingest of the s3 data from LADS, but there is a big delay between sensor time and when it is available in the CMR. The delay seems to vary from 15 days to over 30 days.

Is the automated ingest still helpful if the files coming in are 30 days old?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/USF-IMARS/imars_dags/issues/52#issuecomment-393949098, or mute the thread https://github.com/notifications/unsubscribe-auth/AZUtjQS6rpxrU1a33N09b4CInvhx0yEdks5t4XfUgaJpZM4T-Ddl .

--

Dan Otis, PhDScientific ResearcherUniversity of South FloridaCollege of Marine ScienceInstitute for Marine Remote Sensing (IMaRS)140 7th Ave SouthSt. Petersburg, FL 33701

[dotis]

Search details:

To find data, scroll west on the map and draw a box around the E Caribbean Sea. Then enter “OLCI” in the search box and hit enter.

A list of files should pull up with the most recent files at the top. You are looking for files with “EFR” in the filename. These are full resolution files at 300m.

On Fri, Jun 1, 2018 at 1:31 PM, Daniel Otis dotis@mail.usf.edu wrote:

Maybe for some things, but not for what I need next week. I guess you'll have to go to the visual browser for now. ESA should have an API hub for S3 OLCI data soon.

https://scihub.copernicus.eu/s3/#/home

The login and password are both "s3guest"

On Fri, Jun 1, 2018 at 1:20 PM, Tylar notifications@github.com wrote:

@dotis https://github.com/dotis : so, I set up an automated ingest of the s3 data from LADS, but there is a big delay between sensor time and when it is available in the CMR. The delay seems to vary from 15 days to over 30 days.

Is the automated ingest still helpful if the files coming in are 30 days old?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/USF-IMARS/imars_dags/issues/52#issuecomment-393949098, or mute the thread https://github.com/notifications/unsubscribe-auth/AZUtjQS6rpxrU1a33N09b4CInvhx0yEdks5t4XfUgaJpZM4T-Ddl .

--

Dan Otis, PhDScientific ResearcherUniversity of South FloridaCollege of Marine ScienceInstitute for Marine Remote Sensing (IMaRS)140 7th Ave SouthSt. Petersburg, FL 33701

--

Dan Otis, PhDScientific ResearcherUniversity of South FloridaCollege of Marine ScienceInstitute for Marine Remote Sensing (IMaRS)140 7th Ave SouthSt. Petersburg, FL 33701

[7yl4r]

On top of the heavy delay, the links from CMR fail with "too many redirects". Example link:

https://oceandata.sci.gsfc.nasa.gov/cmr/getfile/S3A_OL_1_EFR____20180505T170719_20180505T171019_20180506T214717_0179_031_012_2340_LN1_O_NT_002.zip

[dotis]

Hmm. I think the S3 is not a priority for NASA at this point.

Were you able to find any good S3 images from today on the ESA visual search hub?

On Mon, Jun 4, 2018 at 3:12 PM, Tylar notifications@github.com wrote:

On top of the heavy delay, the links from CMR fail with "too many redirects". Example link:

https://oceandata.sci.gsfc.nasa.gov/cmr/getfile/S3A_OL_1_ EFR____20180505T170719_20180505T171019_20180506T214717_0179_031012 2340_LN1_O_NT_002.zip

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/USF-IMARS/imars_dags/issues/52#issuecomment-394465969, or mute the thread https://github.com/notifications/unsubscribe-auth/AZUtjZVRLAyFtiy2orh5ApeMoqCuQjZXks5t5YakgaJpZM4T-Ddl .

--

Dan Otis, PhDScientific ResearcherUniversity of South FloridaCollege of Marine ScienceInstitute for Marine Remote Sensing (IMaRS)140 7th Ave SouthSt. Petersburg, FL 33701

[7yl4r]

Not yet. About to do that.

On Mon, Jun 4, 2018, 3:19 PM dotis notifications@github.com wrote:

Hmm. I think the S3 is not a priority for NASA at this point.

Were you able to find any good S3 images from today on the ESA visual search hub?

On Mon, Jun 4, 2018 at 3:12 PM, Tylar notifications@github.com wrote:

On top of the heavy delay, the links from CMR fail with "too many redirects". Example link:

https://oceandata.sci.gsfc.nasa.gov/cmr/getfile/S3A_OL_1_ EFR____20180505T170719_20180505T171019_20180506T214717_0179_031012 2340_LN1_O_NT_002.zip

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub < https://github.com/USF-IMARS/imars_dags/issues/52#issuecomment-394465969>, or mute the thread < https://github.com/notifications/unsubscribe-auth/AZUtjZVRLAyFtiy2orh5ApeMoqCuQjZXks5t5YakgaJpZM4T-Ddl

.

--

Dan Otis, PhDScientific ResearcherUniversity of South FloridaCollege of Marine ScienceInstitute for Marine Remote Sensing (IMaRS)140 7th Ave SouthSt. Petersburg, FL 33701

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/USF-IMARS/imars_dags/issues/52#issuecomment-394467931, or mute the thread https://github.com/notifications/unsubscribe-auth/ABAK_hFPBXpXBCltSEoDI3PKzKqfTsIhks5t5YhSgaJpZM4T-Ddl .

[7yl4r]

Tried getting s3 through copernicus API using the dhus API docs and I think I have it figured out:

1. query for OLCI granules in ROI:

https://scihub.copernicus.eu/s3/api/stub/products?filter=OLCI%20AND%20(%20footprint:%22Intersects(POLYGON((-68.41794442091795%2018.587370193332475,-65.7408430169118%2018.587370193332475,-65.7408430169118%2021.005279979061285,-68.41794442091795%2021.005279979061285,-68.41794442091795%2018.587370193332475)))%22%20)&offset=0&limit=25&sortedby=ingestiondate&order=desc

This returns a list of objects. Excerpts from example object:

{
  "id":1559270, 
  "uuid":"f4ebcb14-49d6-4610-982f-775bbc706415", 
  "identifier":"S3A_OL_1_EFR____20180603T143958_20180603T144258_20180604T190656_0179_032_039_2700_LN1_O_NT_002","footprint":[[[-78.0439,10.8994]${...}, 
  "summary":[
    "Date : 2018-06-03T14:39:57.570059Z",
    "Filename : ${identifier}.SEN3",
    "Identifier : ${identifier}",
    "Instrument : OLCI",
    "Mode : EO",
    "Satellite : Sentinel-3",
    "Size : 604.57 MB"
  ],
  "indexes":[
    {
    "name":"product",
    "children":[
      {"name":"Baseline Collection","value":"002"},
      {"name":"Bright Cover Percentage (%)","value":"4.872840"},
      {"name":"Coastal Cover Percentage (%)","value":"0.000000"},
      {"name":"Cycle number","value":"32"},
      {"name":"ECMWF Type","value":"ANALYSIS"},
      {"name":"Footprint","value":${...}"},
      {"name":"Format","value":"SAFE"},
      {"name":"Fresh Inland Water Cover Percentage (%)","value":"0.000000"},
      {"name":"Ingestion Date","value":"2018-06-04T19:48:13.343Z"},
      {"name":"JTS footprint","value":"POLYGON ((-78.0439 ${...}))"},
      {"name":"Orbit Direction (start)","value":"descending"},
      {"name":"Orbit number (start)","value":"11947"},
      {"name":"PDU Along Track Coordinate (s)","value":"2700"},
      {"name":"PDU Duration (s)","value":"179"},
      {"name":"Pass (start)","value":"23893"},
      {"name":"Pass direction (start)","value":"descending"},
      {"name":"Product level","value":"L1"},
      {"name":"Product type","value":"OL_1_EFR___"},
      {"name":"Relative Orbit Direction (start)","value":"descending"},
      {"name":"Relative orbit (start)","value":"39"},
      {"name":"Relative pass (start)","value":"77"},
      {"name":"Relative pass direction (start)","value":"descending"},
      {"name":"Saline Water Cover Percentage (%)","value":"81.000000"},
      {"name":"Sensing start","value":"2018-06-03T14:39:57.570059Z"},
      {"name":"Sensing stop","value":"2018-06-03T14:42:57.570059Z"},
      {"name":"Tidal Region Cover Percentage (%)","value":"0.000000"},
      {"name":"Timeliness Category","value":"Non Time Critical"}
    ]},
    {"name":"","children":[
        {"name":"Creation Date","value":"2018-06-04T19:06:56.000Z"}]},
        {"name":"summary","children":[
          {"name":"Date","value":"2018-06-03T14:39:57.570059Z"},
          {"name":"Filename","value":"${identifier}.SEN3"},
          {"name":"Identifier","value":"${identifier}"},
          {"name":"Instrument","value":"OLCI"},
          {"name":"Mode","value":"EO"},
          {"name":"Satellite","value":"Sentinel-3"},
          {"name":"Size","value":"604.57 MB"}
    ]},{
      "name":"instrument","children":[
        {"name":"Instrument abbreviation","value":"OLCI"},
        {"name":"Instrument mode","value":"Earth Observation"},
        {"name":"Instrument name","value":"Ocean Land Colour Instrument"}
    ]},{"name":"platform","children":[
        {"name":"Mission type","value":"Earth observation"},
        {"name":"NSSDC identifier","value":"2016-011A"},
        {"name":"Operator","value":"European Space Agency"},
        {"name":"Satellite description","value":"\u003ca target\u003d\"_blank\" href\u003d\"https://sentinel.esa.int/web/sentinel/missions/sentinel-3\"\u003ehttps://sentinel.esa.int/web/sentinel/missions/sentinel-3\u003c/a\u003e"},
        {"name":"Satellite name","value":"Sentinel-3"},{"name":"Satellite number","value":"A"}
      ]},{"name":"processing","children":[
        {"name":"Online Quality Check","value":"PASSED"},
        {"name":"Processing Facility Name","value":"Land OLCI Processing and Archiving Centre [LN1]"},
        {"name":"Processing Facility Organization","value":"European Space Agency"},
        {"name":"Processing level","value":"1"},
        {"name":"Processing name","value":"Data Processing"}  
      ]
  }],
  "thumbnail":true,
  "quicklook":true,
  "instrument":"OLCI",
  "productType":"OL_1_EFR___",
  "itemClass":"http://www.esa.int/s3#ipf_olci_level_1"
}

So there's some good info there. Then we just fetch the product using the uuid

tylar@tylardesk:~/s3test$ wget --user=s3guest --password=*******  'https://scihub.copernicus.eu/s3/odata/v1/Products('\''f4ebcb14-49d6-4610-982f-775bbc706415'\'')/$value'
--2018-06-05 15:44:58--  https://scihub.copernicus.eu/s3/odata/v1/Products('f4ebcb14-49d6-4610-982f-775bbc706415')/$value
Resolving scihub.copernicus.eu (scihub.copernicus.eu)... 131.176.236.28
Connecting to scihub.copernicus.eu (scihub.copernicus.eu)|131.176.236.28|:443... connected.
HTTP request sent, awaiting response... 401 Unauthorized
Authentication selected: Basic realm="OData service"
Reusing existing connection to scihub.copernicus.eu:443.
HTTP request sent, awaiting response... 200 OK
Length: 633948030 (605M) [application/octet-stream]
Saving to: ‘$value’

$value                             0%[                                                          ]   2.80M   754KB/s    eta 12m 42s^C

I didn't find a dhus python client, but I should be able to whip up a few quick scripts based on this.

[dotis]

Ok. Sounds good. From what I've read the uuid is the best way to grab files.

On Tue, Jun 5, 2018 at 11:59 AM, Tylar notifications@github.com wrote:

Tried getting s3 through copernicus API using the dhus API docs https://scihub.copernicus.eu/twiki/do/view/SciHubUserGuide/5APIsAndBatchScripting and I think I have it figured out:

1. query for OLCI granules in ROI:

https://scihub.copernicus.eu/s3/api/stub/products?filter= OLCI%20AND%20(%20footprint:%22Intersects(POLYGON((-68.41794442091795%2018. 587370193332475,-65.7408430169118%2018.587370193332475,-65. 7408430169118%2021.005279979061285,-68.41794442091795%2021. 005279979061285,-68.41794442091795%2018.587370193332475)))%22%20)& offset=0&limit=25&sortedby=ingestiondate&order=desc

This returns a list of objects. Excerpts from example object:

{ "id":1559270, "uuid":"f4ebcb14-49d6-4610-982f-775bbc706415", "identifier":"S3A_OL_1_EFR__20180603T143958_20180603T144258_20180604T190656_0179_032_039_2700_LN1_O_NT_002","footprint":[[[-78.0439,10.8994]${...}, "summary":[ "Date : 2018-06-03T14:39:57.570059Z", "Filename : ${identifier}.SEN3", "Identifier : ${identifier}", "Instrument : OLCI", "Mode : EO", "Satellite : Sentinel-3", "Size : 604.57 MB" ], "indexes":[ { "name":"product", "children":[ {"name":"Baseline Collection","value":"002"}, {"name":"Bright Cover Percentage (%)","value":"4.872840"}, {"name":"Coastal Cover Percentage (%)","value":"0.000000"}, {"name":"Cycle number","value":"32"}, {"name":"ECMWF Type","value":"ANALYSIS"}, {"name":"Footprint","value":${...}"}, {"name":"Format","value":"SAFE"}, {"name":"Fresh Inland Water Cover Percentage (%)","value":"0.000000"}, {"name":"Ingestion Date","value":"2018-06-04T19:48:13.343Z"}, {"name":"JTS footprint","value":"POLYGON ((-78.0439 ${...}))"}, {"name":"Orbit Direction (start)","value":"descending"}, {"name":"Orbit number (start)","value":"11947"}, {"name":"PDU Along Track Coordinate (s)","value":"2700"}, {"name":"PDU Duration (s)","value":"179"}, {"name":"Pass (start)","value":"23893"}, {"name":"Pass direction (start)","value":"descending"}, {"name":"Product level","value":"L1"}, {"name":"Product type","value":"OL_1EFR"}, {"name":"Relative Orbit Direction (start)","value":"descending"}, {"name":"Relative orbit (start)","value":"39"}, {"name":"Relative pass (start)","value":"77"}, {"name":"Relative pass direction (start)","value":"descending"}, {"name":"Saline Water Cover Percentage (%)","value":"81.000000"}, {"name":"Sensing start","value":"2018-06-03T14:39:57.570059Z"}, {"name":"Sensing stop","value":"2018-06-03T14:42:57.570059Z"}, {"name":"Tidal Region Cover Percentage (%)","value":"0.000000"}, {"name":"Timeliness Category","value":"Non Time Critical"} ]}, {"name":"","children":[ {"name":"Creation Date","value":"2018-06-04T19:06:56.000Z"}]}, {"name":"summary","children":[ {"name":"Date","value":"2018-06-03T14:39:57.570059Z"}, {"name":"Filename","value":"${identifier}.SEN3"}, {"name":"Identifier","value":"${identifier}"}, {"name":"Instrument","value":"OLCI"}, {"name":"Mode","value":"EO"}, {"name":"Satellite","value":"Sentinel-3"}, {"name":"Size","value":"604.57 MB"} ]},{ "name":"instrument","children":[ {"name":"Instrument abbreviation","value":"OLCI"}, {"name":"Instrument mode","value":"Earth Observation"}, {"name":"Instrument name","value":"Ocean Land Colour Instrument"} ]},{"name":"platform","children":[ {"name":"Mission type","value":"Earth observation"}, {"name":"NSSDC identifier","value":"2016-011A"}, {"name":"Operator","value":"European Space Agency"}, {"name":"Satellite description","value":"\u003ca target\u003d\"_blank\" href\u003d\"https://sentinel.esa.int/web/sentinel/missions/sentinel-3\"\u003ehttps://sentinel.esa.int/web/sentinel/missions/sentinel-3\u003c/a\u003e http://sentinel.esa.int/web/sentinel/missions/sentinel-3%5Cu003c/a%5Cu003e"}, {"name":"Satellite name","value":"Sentinel-3"},{"name":"Satellite number","value":"A"} ]},{"name":"processing","children":[ {"name":"Online Quality Check","value":"PASSED"}, {"name":"Processing Facility Name","value":"Land OLCI Processing and Archiving Centre [LN1]"}, {"name":"Processing Facility Organization","value":"European Space Agency"}, {"name":"Processing level","value":"1"}, {"name":"Processing name","value":"Data Processing"} ] }], "thumbnail":true, "quicklook":true, "instrument":"OLCI", "productType":"OL_1_EFR___", "itemClass":"http://www.esa.int/s3#ipf_olci_level_1" }

So there's some good info there. Then we just fetch the product using the uuid https://scihub.copernicus.eu/s3/odata/v1/Products('5d76e7a2-621e-498f-bc23-f52da4072544')/$value

tylar@tylardesk:~/s3test$ wget --user=s3guest --password=*** 'https://scihub.copernicus.eu/s3/odata/v1/Products('\''f4ebcb14-49d6-4610-982f-775bbc706415'\'')/$value' --2018-06-05 15:44:58-- https://scihub.copernicus.eu/s3/odata/v1/Products('f4ebcb14-49d6-4610-982f-775bbc706415')/$value Resolving scihub.copernicus.eu (scihub.copernicus.eu)... 131.176.236.28 Connecting to scihub.copernicus.eu (scihub.copernicus.eu)|131.176.236.28|:443... connected. HTTP request sent, awaiting response... 401 Unauthorized Authentication selected: Basic realm="OData service" Reusing existing connection to scihub.copernicus.eu:443. HTTP request sent, awaiting response... 200 OK Length: 633948030 (605M) [application/octet-stream] Saving to: ‘$value’

$value 0%[ ] 2.80M 754KB/s eta 12m 42s^C

I didn't find a dhus python client, but I should be able to whip up a few quick scripts based on this.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/USF-IMARS/imars_dags/issues/52#issuecomment-394764891, or mute the thread https://github.com/notifications/unsubscribe-auth/AZUtjfBaheXRe8ht2i8OZNFtX_Q6V-MNks5t5qregaJpZM4T-Ddl .

--

Dan Otis, PhDScientific ResearcherUniversity of South FloridaCollege of Marine ScienceInstitute for Marine Remote Sensing (IMaRS)140 7th Ave SouthSt. Petersburg, FL 33701

[7yl4r]

Getting there on this one. The latest files are pulling through dhus into /srv/imars-objects/gom/s3a_ol_1_efr/, but the dates on the files & in the database are wrong.