search

USGS-CMG / usgs-cmg-portal

do_convert.sh
6 stars 13 forks source link

CMG Time Series data not appearing in East Coast Historical Sensor Catalog #117

Closed rsignell-usgs closed 8 years ago

rsignell-usgs commented 9 years ago

I selected the East Coast Historical Sensor Catalog, then "USGS Coastal and Marine Program" and "turbidity" and got no results: 2015-10-23_0636

I then chose USGS Time Series Database, , then "USGS Coastal and Marine Program" and "sea water turbidity" and got 26 results: 2015-10-23_0635

Does this mean that some or all of the USGS Time Series Database stations have not yet been integrated in the East Coast Historical Sensor Data?

I find this confusing.

kwilcox commented 9 years ago

It's just an artifact from the previous incarnation of the Time Series Database, I'll add the new incarnation to the East Coast module and they will function how you expect.

akbstone commented 9 years ago

Sea Water Turbidity is now displaying.

kwilcox commented 9 years ago

@rsignell-usgs all set to close?

rsignell-usgs commented 8 years ago

Indeed, sea water turbidity is now displaying: 2016-02-02_6-00-21

But @emontgomery-usgs , this spike, with value of 939. is not a missing data value, right?
This should be eliminated from the original data, shouldn't it? Can you investigate?

emontgomery-usgs commented 8 years ago

Yes, that is a spike in turbidity, not a missing or fill value. Depending on the processing, they may be left intentionally. Turbidity may have what look like random spikes, but that are really really data, so some actual spikes probably crept through in the interests of not removing "good" data.

On Tue, Feb 2, 2016 at 6:05 AM, Rich Signell notifications@github.com wrote:

Indeed, sea water turbidity is now displaying: [image: 2016-02-02_6-00-21] https://cloud.githubusercontent.com/assets/1872600/12747558/b26db642-c972-11e5-8ec1-300841ef6f7d.png

But @emontgomery-usgs https://github.com/emontgomery-usgs , this spike, with value of 939. is not a missing data value, right?

This should be eliminated from the original data, shouldn't it? Can you investigate?

— Reply to this email directly or view it on GitHub https://github.com/USGS-CMG/usgs-cmg-portal/issues/117#issuecomment-178511778 .

Ellyn Montgomery, Oceanographer and Data Manager U.S. Geological Survey Woods Hole Coastal and Marine Science Center 384 Woods Hole Rd., Woods Hole, MA, 02543-1598 (508) 457-2356

emontgomery-usgs commented 8 years ago

I fixed this spike, and replaced the file in CF-1.6/WFAL on geoport. If I understand Kyle's explanation, it will get re-harvested automatically, and look nicer next time it's examined.

On Tue, Feb 2, 2016 at 8:41 AM, Montgomery, Ellyn emontgomery@usgs.gov wrote:

Yes, that is a spike in turbidity, not a missing or fill value. Depending on the processing, they may be left intentionally. Turbidity may have what look like random spikes, but that are really really data, so some actual spikes probably crept through in the interests of not removing "good" data.

On Tue, Feb 2, 2016 at 6:05 AM, Rich Signell notifications@github.com wrote:

Indeed, sea water turbidity is now displaying: [image: 2016-02-02_6-00-21] https://cloud.githubusercontent.com/assets/1872600/12747558/b26db642-c972-11e5-8ec1-300841ef6f7d.png

But @emontgomery-usgs https://github.com/emontgomery-usgs , this spike, with value of 939. is not a missing data value, right?

This should be eliminated from the original data, shouldn't it? Can you investigate?

— Reply to this email directly or view it on GitHub https://github.com/USGS-CMG/usgs-cmg-portal/issues/117#issuecomment-178511778 .

Ellyn Montgomery, Oceanographer and Data Manager U.S. Geological Survey Woods Hole Coastal and Marine Science Center 384 Woods Hole Rd., Woods Hole, MA, 02543-1598 (508) 457-2356

Ellyn Montgomery, Oceanographer and Data Manager U.S. Geological Survey Woods Hole Coastal and Marine Science Center 384 Woods Hole Rd., Woods Hole, MA, 02543-1598 (508) 457-2356

rsignell-usgs commented 8 years ago

Hmm, I just tried this (3:56PM ET) and we still have the problem.
@kwilcox, I thought the data is reharvested every 3 hours, so shouldn't the spike be gone?

emontgomery-usgs commented 8 years ago

I was also wondering about the re-harvest time.

There is new 2014 Barnegat data added yesterday that isn't showing up yet either. Similarly, the locations that show up on the index map in Worcester MA, and Sacramento CA were corrected, so shouldn't be there anymore. (at least hopefully not)

On Thu, Feb 4, 2016 at 4:01 PM, Rich Signell notifications@github.com wrote:

Hmm, I just tried this (3:56PM ET) and we still have the problem.

@kwilcox https://github.com/kwilcox, I thought the data is reharvested every 3 hours, so shouldn't the spike be gone?

— Reply to this email directly or view it on GitHub https://github.com/USGS-CMG/usgs-cmg-portal/issues/117#issuecomment-180047236 .

Ellyn Montgomery, Oceanographer and Data Manager U.S. Geological Survey Woods Hole Coastal and Marine Science Center 384 Woods Hole Rd., Woods Hole, MA, 02543-1598 (508) 457-2356

rsignell-usgs commented 8 years ago

@emontgomery-usgs, it turns out that although our CSW catalog is harvested every 3 hours, harvest of our time series data is not on a schedule. It is only harvested when @kwilcox manually does it.