search

CMU-CREATE-Lab / data-visualization-tools

EarthTime, and various data visualization libraries
Other
14 stars 5 forks source link

Landsat Base Layer Not Updating in New Waypoints (wef-prod) #228

Closed andrewcberkley closed 3 years ago

andrewcberkley commented 3 years ago

Hi All,

I hope you're doing well! Seems like the US is starting to get back to a sense of normalcy, while it's unfortunately very much a different story for us here in Europe still waiting to be vaccinated.

I've generally been diligent in keeping stories up-to-date with the most recent Landsat imagery and just noticed that I didn't do it for the original Surface Water story. I let the waypoint generate, but even though the end date is set to 20191232, it's stopping at 2016:

https://wef-prod.earthtime.org/m/stories/Surface_Water

https://earthtime.org/#v=44.53441,59.27268,5.62,latLng&t=1.41&ps=50&bt=19840101&et=20191231&l=blsat&startDwell=1&endDwell=3

I think this might be a wef-prod (link to the sheet) specific issue related to Issue #197 and old codebases; however, in some of the other stories I've updated in wef-prod, the Landsat base layer seems to be fine:

https://wef-prod.earthtime.org/m/stories/Deforestation

https://earthtime.org/#v=-17.27631,-62.10121,11.261,latLng&t=1.41&ps=50&bt=19840101&et=20191231&l=blsat&startDwell=1&endDwell=3

Not really sure what the issue is, but regardless maybe we could look at doing a general update of wef-prod when you all have a free moment.

Take care!

-Andrew

jjkohler commented 3 years ago

https://staging.earthtime.org/explore#csvlayers=1tV1xc41BS92JAtL49Dv27c4UjnLlVJm6O0EM3dVvbFo.0&waypoints=1tV1xc41BS92JAtL49Dv27c4UjnLlVJm6O0EM3dVvbFo.2077947989

Hi @andrewcberkley, sorry for the slow response to your issue here. We recommend that the best way to address the issue you've been facing is probably going to be to get your earthtime instance updated to our latest codebase. At the above link (using the regular ET username/pw) you should be able to demo the functionality of your layers/stories in the new codebase. I made a copy of your sheets and brought them up to match the sheet format that is currently in use for new sheets. I think this will include some columns that will be helpful for you going forward - particularly the ability to specify specific views for mobile stories in the waypoints sheet. I recommend that you review this version of ET and make sure that there's no critical regressions that would impact your work. Once you're comfortable with it, @pdille can switch your domain out to the latest code, and this should address the landsat issues you're having, among other things.

andrewcberkley commented 3 years ago

Hi @jjkohler, no worries at all! Thanks so much for taking the time to look into it.

Just tried the staging environment for the new codebase and did some spot checks on stories. While everything seems fine in explore/desktop version, when switching to mobile, it seems like the landsat issue unfortunately persists within the stories:

https://staging.earthtime.org/m/stories/surface_water#csvlayers=1tV1xc41BS92JAtL49Dv27c4UjnLlVJm6O0EM3dVvbFo.0&waypoints=1tV1xc41BS92JAtL49Dv27c4UjnLlVJm6O0EM3dVvbFo.2077947989

Not sure if this is a matter of it being within the staging environment or something else, but as long as the mobile view is good to go, happy for you and @pdille to switch the domain!

jjkohler commented 3 years ago

Thanks for highlighting this issue. We have now been able to identify the issue that was causing these stories to not pull in the more recent landsat data. The issue should be fixed at the staging site link sent previously. Stories previously rendered and cached with the old landsat will need reloaded. The current method for this is still to add a suffix onto the waypoint link, and reload the story. I did this for the story you mentioned in your last comment, and it looks like it's all loading correctly now.

andrewcberkley commented 3 years ago

Hi @jjkohler, thanks so much for looking into this--very much appreciated!

Just did another quick spot check and everything seems to be fine with the mobile stories; however, on the desktop version, it doesn't seem like some stories are loading the layers when in story mode (worth noting that they load correctly when adding them in the data library when in explore mode, though). Here's a few of the ones I noticed with issues:

With that being said, given that we only display the mobile version more than happy to move forward with you and @pdille pushing the more recent codebase to "wef-prod".

pdille commented 3 years ago

Hey @andrewcberkley . Thanks for your patience with this. Looks like there were some column ordering issues with the spreadsheet. The 3 links you made note of seem to be working now. Please test and any others just to make sure everything is truly as you expect.

Thanks!

andrewcberkley commented 3 years ago

Hi @pdille, thanks so much for looking into all of the issues and getting things up-to-date! We just finished checking a significant number of stories and everything seems good to go! Ready for you to flip the switch!

pdille commented 3 years ago

@andrewcberkley Great to hear. A question we do have though is around the second link in your earlier post:

https://staging.earthtime.org/stories/the_tipping_point#csvlayers=1tV1xc41BS92JAtL49Dv27c4UjnLlVJm6O0EM3dVvbFo.0&waypoints=1tV1xc41BS92JAtL49Dv27c4UjnLlVJm6O0EM3dVvbFo.2077947989

That layer from the Climate Impact Lab seems to be quite different from what was previous rendered out for thumbnails. Has this layer changed in the interim on your end or has the latest codebase caused some issue parsing it?

andrewcberkley commented 3 years ago

Uh-oh, the layers haven't changed, so I'm guessing it's a parsing issue. The mobile view still seems fine, but within the explore view, it's not rendering the primary layer. Thanks for catching this!

pdille commented 3 years ago

Hmm... what's odd though is that the geojson it is referencing (from the spreadsheet) matches what we see:

https://tiles.earthtime.org/aberkley/data/costs_of_climate_change/BRA.4.R9fd5b7ed1bcf96a7.geojson

andrewcberkley commented 3 years ago

Hi @pdille that's odd indeed! The geojson "BRA_4_R9fd5b7ed1bcf96a7" is more of a secondary layer that covers up the few motheaten spots from the primary layer in this story, "50_percent_probability_RCP85_tas_seasonal_JJA" in the wef-prod CSV Layers sheet (which originally derives from the "world_counties_test" in the Default CSV Layers sheet that @arwright so kindly created for this story due to the level of detail and complexity of the geojson). Not really sure why both layers were displaying earlier in the week and only one layer displaying now.

Not sure if it's of any significance, but just wanted to also flag that in the tiles directory you shared, all of the layer's underscores have been converted to periods.

pdille commented 3 years ago

I think I see what is going on here.

The issue is that the latest codebase is strict in how it interprets layer names and the longstanding issue of dashes and underscores is at play here. The layer IDs used in the share links for the waypoints in question are referring to layers with strictly underscores, whereas some of the layer IDs in the CSV definition sheet have dashes. The system therefore can't find the layer the waypoint is asking for.

e.g.: CSV layer definition: 50_percent_probability_RCP85_tas-seasonal-JJA Waypont sharelink layer reference: 50_percent_probability_RCP85_tas_seasonal_JJA

In our WIP wef CSV Layers tab, I made the change to have all layer definitions use underscore, so staging should now load all these layers again in the stories.

e.g. https://staging.earthtime.org/stories/the_tipping_point#csvlayers=1tV1xc41BS92JAtL49Dv27c4UjnLlVJm6O0EM3dVvbFo.0&waypoints=1tV1xc41BS92JAtL49Dv27c4UjnLlVJm6O0EM3dVvbFo.2077947989

I think this may be the last of the issues to get you all using the latest codebase. I do see you edited your wef-prod sheet since we forked the sheet for testing, so we'll test and then bring over these changes. Let us know a good day to make this switch over, so as to not break anything important you all may be in the middle of.

pdille commented 3 years ago

Going forward, I would suggest no longer uses dashes in the ID of a layer, though you still technically can (just ensure everything matches exactly) The old system was lenient in reading in waypoint layer ids, but this is no longer the case.

andrewcberkley commented 3 years ago

Hi @pdille, thanks so much! Happy for you to do the switch at the earliest convivence since we don't have anything major coming up anytime soon. Going forward, we'll try not to use any dashes in the layer IDs (we're generally underscore people, so not sure why that happened in a layer where the dash follows a few earlier underscores), but if we run into errors in the new codebase, we'll check that during our troubleshooting process before raising an issue.

pdille commented 3 years ago

@andrewcberkley

Finally turned the key to bring you all up to the latest codebase and spreadsheet format. I think everything is working, but please let us know.

Note that the format of the sheets (waypoints and layers) has changed slightly, so if you have code that auto generates the rows, you will need to modify.

Otherwise, I think I ensured your latest additions to the sheets from this week were carried over, but apologizes if not. Both the original sheets were backed up incase we need to revert. You'll see their tabs with a 'backup' prefix.

Let us know how things look and it may be worth for either Jared or I to talk sometime about the new things you can do.

andrewcberkley commented 3 years ago

Amazing! Thank you so much @pdille! We just did a spot check of some stories and everything seems to be running smoothly! I'll try to set some time up with either you or @jjkohler soon so we can discuss the new features. Thanks again!