2 new datasets to harvest to portal

[emontgomery-usgs]

@kwilcox , @rsignell-usgs There are 2 new datasets on our server ready for harvest: http://geoport.whoi.edu/thredds/catalog/silt/usgs/Projects/stellwagen/CF-1.6/W_GMAINE/catalog.html http://geoport.whoi.edu/thredds/catalog/silt/usgs/Projects/stellwagen/CF-1.6/GRANDBAY/catalog.html

Please ingest them when you have the chance. Thanks!

[rsignell-usgs]

@kwilcox , this needs your attention.

[emontgomery-usgs]

and while looking near the gulf coast, I noticed these 2 experiments had also escaped ingest: http://geoport.whoi.edu/thredds/dodsC/silt/usgs/Projects/stellwagen/CF-1.6/CHANDELEUR_13/catalog.html and http://geoport.whoi.edu/thredds/dodsC/silt/usgs/Projects/stellwagen/CF-1.6/DAUPHIN/catalog.html thanks!

[emontgomery-usgs]

and if you want to try to ingest a sample dataset from the Santa Cruz office, I was able to convert this to a nominally CF output: https://gamone.whoi.edu/thredds/silt/usgs/Projects/stellwagen/Data_tests/tst_other_centers/catalog.html Cheers!

[emontgomery-usgs]

@kwilcox , I see that the data from CHANDELEUR_13 and DAUPHIN got ingested, thanks! I'm still not seeing the original 2 I asked about though (W_GMAINE and GRANDBAY)- is there something wrong with theses files? Thanks!

[emontgomery-usgs]

@kwilcox, any chance you could get these 2 experiments harvested to the portal this week? http://geoport.whoi.edu/thredds/catalog/silt/usgs/Projects/stellwagen/CF-1.6/W_GMAINE/catalog.html http://geoport.whoi.edu/thredds/catalog/silt/usgs/Projects/stellwagen/CF-1.6/GRANDBAY/catalog.html Thanks!

[kwilcox]

Looking now!

[kwilcox]

Made some good progress, having it looked over internally and I'll post a link for you to review once that is done.

I switched our import to directly hit your THREDDS files. We were previously downloading the Raw and converting to CF and then to our internal Device files. We are no longer producing the Device files and are harvesting directly from your the CF files - reducing a bunch of complexity and opening up the ability to harvest from other THREDDS endpoints of CF files (i.e. Santa Cruz files).

[emontgomery-usgs]

FANTASTIC! thanks!

[emontgomery-usgs]

@kwilcox - Any guesses on when these 2 might be on the portal? Doesn't have to be auto-harvested... thanks!

[kwilcox]

It’s on top of the checklist this week

[kwilcox]

Data is up on staging: https://cmgdata.usgsportals.net/?&staging=true&sensor_version=v2#metadata/86/sensor_source/list

Could you spot check?

[emontgomery-usgs]

@kwilcox ,

I see the data from both new datasets- great! But now I can't find the control to show all data instead of whatever approximation is shown by default. All the data I tried to look at appears to be WAY over-simplified in the default view. Thanks!

[kwilcox]

Thanks, I noticed this as well and pinged @akbstone to take a look. It is likely because this is the staging instance and isn't completely polished for production. I've downloaded the CSV from the stations and the data looks good so if isn't a quick fix I'll just push to production.

[emontgomery-usgs]

OK, thanks.

Another detail is that I'm not sure all the stations for all the experiments are shown: I think they will likely all be displayed when pushed to the existing portal.

[kwilcox]

You are on the W_GMAINE - 1055 page and it is only showing stations within a certain geographic area around that station.

The catalog page for all of the stations shows them all: https://cmgdata.usgsportals.net/?&staging=true&sensor_version=v2#metadata/86/sensor_source/list

If that is confusing maybe we could add a note that the map is only showing nearby stations, or disable the map controls on the station page?

[akbstone]

@emontgomery-usgs @kwilcox - the binning issue is fixed.

[akbstone]

@emontgomery-usgs @kwilcox

Re. nearby stations on station page, I think a couple things could make this more usable:

• Limit zoom to the selected zoom and higher
• On map pan, query for stations within the new bounds

[kwilcox]

Both sound great to me!

[emontgomery-usgs]

@akbstone , @kwilcox I'd agree- both would help - thanks!

In the last image I put in this thread, there are a ton of other stations from other sources, so I assumed the map would also show everything of ours in the view. Couldn't find out how to remove "other" stations from the view- if there's a way to do that, it would be nice if it were more obvious.

I submitted a new ticket #302 suggesting use of color instead of shape to indicate the site. When dots are dense at certain zooms, it was hard to separate the star from all the dots.

[emontgomery-usgs]

@kwilcox , @akbstone, Could the data from the 2 new experiments be pushed to the public server some time soon?

Also I went to the link to the staging server in Kyle's post just now and it just cycles and cycles... and doesn't ever return content. Thanks!

[kwilcox]

@jessicaaustin and I will push these to the portal today, it's happening!

[akbstone]

@emontgomery-usgs @kwilcox

• This link is now loading - sorry about the trouble: https://cmgdata.usgsportals.net/?&staging=true&sensor_version=v2#metadata/86/sensor_source/list
• I made a couple changes to how the large station star is displayed.. it's now larger and always shows up on top of all other stations present. https://cmgdata.usgsportals.net/?&staging=true&sensor_version=v2#metadata/57090/station/data

[emontgomery-usgs]

@akbstone Thank you! much better!

[kwilcox]

@emontgomery-usgs https://cmgdata.usgsportals.net/?&sensor_version=v2#metadata/86/sensor_source/list

The data is streaming in and will be updating into the portal, some stations will be viewable while some may take a few hours. Searching for individual stations may not work for a few hours since a reindex won't happen until all data is in there.

Also, this is using the new ingestion system so we are all good looking forward!

[emontgomery-usgs]

@kwilcox

Wonderful, thanks! So excited the new ingest system is in place!

If I follow your link above, I see the new datasets in the list. However, if I go to the Main portal page, add the Oceanographic timeseries data to the map, and then zoom to where the Grand Bay data should be, the stations are not displayed. The sites show up on the map on the station page, but not on the map in the data browser. Is this intended?

[emontgomery-usgs]

@kwilcox @akbstone The 2 new datasets don't show up in the map view. You can't find them if you're browsing the map, even though they're there if you know how to find the station page.

Thanks for any help!

[kwilcox]

@emontgomery-usgs Brian and I took a look and figured out what was happening.

You can see them here: https://cmgdata.usgsportals.net/?&sensor_version=v2&ls=793b54c9-6055-1fd2-2eba-13ddec139891#map

But notice the "sensor_version=v2" in the URL. Without that are going to be missing from the map because they are on the "new system" and "V2" combines both systems together. The solution is to make "sensor_version=v2" the default which is only held up by a bug in displaying curtain plots (hover isn't working). Brian needs some more time to work that into his schedule, lets touch base mid next week.

[emontgomery-usgs]

@kwilcox, thanks for looking into this. Sounds like you and Brian have made progress. Great! Keep up the good work and I'll ping you next week, if I haven't heard by Thurs.

[emontgomery-usgs]

@kwilcox, OK, here's your reminder. Today I don't see the Western Gulf of Maine, or Grand Bay (Louisiana) data at https://cmgdata.usgsportals.net/#metadata/86/sensor_source/list or https://cmgdata.usgsportals.net/#map. Has the v2 merge happened? Thanks!

[emontgomery-usgs]

@kwilcox, @dnowacki-usgs, @rsignell-usgs

Now we're up to 4 experiments of data thata haven't made it to the portal. Is there any chance that they could be added before I retire on 7/31? I'm trying to tie up loose ends, and it would really be great to have the last four of our data releases visible before I leave :> If you could do what needs to be done to get it onto the IOOS porta,l bypassing the USGS portal, that would be fine too.

Here are the links to the data that is not yet on either portal: http://geoport.whoi.edu/thredds/catalog/silt/usgs/Projects/stellwagen/CF-1.6/W_GMAINE/catalog.html http://geoport.whoi.edu/thredds/catalog/silt/usgs/Projects/stellwagen/CF-1.6/GRANDBAY/catalog.html http://geoport.whoi.edu/thredds/catalog/silt/usgs/Projects/stellwagen/CF-1.6/NRPP_17/catalog.html http://geoport.whoi.edu/thredds/catalog/silt/usgs/Projects/stellwagen/CF-1.6/MATANZAS/catalog.html

Many thanks!