search

TerriaJS / nationalmap

Australia's NationalMap
https://nationalmap.gov.au
122 stars 45 forks source link

NOM 2022 Constraint Map Update #1153

Closed elangham closed 2 years ago

elangham commented 2 years ago

Annual update of ISF/UTS data for the Network Opportunity Maps layers. Standard process requires:

1) ISF sending Data61 the data files (links provided below) 2) Data61 rebuilding the polygons and loading on a test site 3) ISF confirming all looks/works as expected or noting issues 4) Shipping to production

Four things to note:

![Uploading Screen Shot 2022-06-03 at 4.42.20 pm.png…]()

GEOJSON FILE LINK https://network-opportunity-maps.s3.ap-southeast-2.amazonaws.com/constraints/published/available_capacity.geojson

CSV FILE LINKS (called on by the geojson) https://network-opportunity-maps.s3.ap-southeast-2.amazonaws.com/constraints/published/proposed_investment.csv https://network-opportunity-maps.s3.ap-southeast-2.amazonaws.com/constraints/published/peak_day.csv https://network-opportunity-maps.s3.ap-southeast-2.amazonaws.com/constraints/published/deferral_values_timeseries.csv https://network-opportunity-maps.s3.ap-southeast-2.amazonaws.com/constraints/published/available_capacity_timeseries.csv

Thanks guys

nf-s commented 2 years ago

I have updated CSV URLS

We're making a couple of minor edits to the moustache template but I think we control these now so this shouldn't need to involve you.

Currently, we are controlling everything in the NationalMap "Energy" group. Let me know what changes you would like me to make and I can publish them 🙂

We need to change the order of files in the public Network Opportunities directory - it should be as per the attached image.

I am unable to see the screenshot, I don't think it uploaded properly

Data61 rebuilding the polygons and loading on a test site

Working on that now

elangham commented 2 years ago

Screen Shot 2022-06-03 at 4 42 20 pm

Does this screenshot reuploading work @nf-s ?

nf-s commented 2 years ago

@elangham I can see it now - thanks!

nf-s commented 2 years ago

Hi @elangham

I have updated regions and re-ordered items in the "Network Opportunities"

When you have a chance can you please verify my changes:

Cheers!

nf-s commented 2 years ago
elangham commented 2 years ago

Thanks @nf-s - are you able to add my colleague Jaysson Guerrero who will review - his username is @jego9130 - I don't seem to be able to add a mention presumably as he's not listed on the team?

nf-s commented 2 years ago

I think GitHub mentions are playing up a bit - I had trouble getting it to recognise your username.

It seems to be working now though

image

elangham commented 2 years ago

I can't get it to work but it seems like he can see it anyway - thanks!

jego9130 commented 2 years ago

Hi @nf-s

I’ve reviewed the maps in the test link, and it looks like the polygon rebuild is ok. However, it seems that there are some moustache template issues. We’re unsure if that is on your end or ours. I've attached a doc with the identified issues and some examples.

Thanks!

Terra_site_comments.pdf

nf-s commented 2 years ago

Hi @jego9130

Sorry I forgot to say that the test link won't have CSS for feature info templates

Test link with CSS

image image

I will look into the "investment year" legend issue.

I have also noticed a discrepancy between the custom legend images and symbology - there may be an issue with how the binning is being done - I will investigate and get back to you.

jego9130 commented 2 years ago

Thanks for the update! One question, I clicked on the 'Test link with CSS' but it sent to the national map with the old data. Actually, it looks like the 'proposed investment' layer has the new data, but the other layers have the old data. Is there a link with CSS with the new data? Sorry, I'm a bit confused.

nf-s commented 2 years ago

HI @jego9130

Sorry I should have been more clear.

The first test links - will show everything except for custom CSS for featureInfoTemplate

The second link - will show correct custom CSS, with new dataset (CSV) but using old regions

Sorry for the confusion, this is because the CSS is hardcoded in nationalmap deployment - and therefore isn't available in the first test links. But, due to how the "test links" are constructed - they aren't able to be used on production maps.

Long story short, the first links are the ones to use - but you won't be able to see CSS until they updated in nationalmap

nf-s commented 2 years ago

Update on discrepancy between custom legend image and symbology:

I can confirm that the symbology was not matching the legend image. The left of the image is what it should be, the right we have at the moment.

image

While fixing that I ran into a bug where I can't provide a custom legend image for discrete/binned styles. Are you happy with the Terria-generated legend? (in the left of the previous image)

Update on mustache template

There is definitely a bug which is causing certain values to not show up - https://github.com/TerriaJS/terriajs/issues/6228 - I am investigating

nf-s commented 2 years ago

Ticket to add support for multiple chart lines in Feature Info - https://github.com/TerriaJS/terriajs/issues/6434

jego9130 commented 2 years ago

Update on discrepancy between custom legend image and symbology:

I can confirm that the symbology was not matching the legend image. The left of the image is what it should be, the right we have at the moment.

image

While fixing that I ran into a bug where I can't provide a custom legend image for discrete/binned styles. Are you happy with the Terria-generated legend? (in the left of the previous image)

Update on mustache template

There is definitely a bug which is causing certain values to not show up - TerriaJS/terriajs#6228 - I am investigating

Thanks for claryfing the links confusion and fixing the legend issue. I think it looks ok.

jego9130 commented 2 years ago

@nf-s , you're still investigating the mustache template and chart issue, right?

nf-s commented 2 years ago

Hi @jego9130 yes working on those issues as we speak. Will hopefully have a PR in the next couple of days

jego9130 commented 2 years ago

Ok thanks for the update.

jego9130 commented 2 years ago

Hi @nf-s, just touching base on this. Have you been able to fix those issues? Thanks

AnaBelgun commented 2 years ago

We will have a bit of a delay @jego9130. We'll let you know as soon as we come back to this issue/PR. Apologies.

AnaBelgun commented 2 years ago

Dependent on https://github.com/terriajs/terriajs/issues/6441

nf-s commented 2 years ago

New test link - this fixes mustache template issue - note this link will not have custom CSS

AnaBelgun commented 2 years ago

@jego9130 please test the link provided by Nick above and let us know of the results.

jego9130 commented 2 years ago

Thanks @nf-s and @AnaBelgun. I've reviewed the link and many issues have been fixed. However, I've noticed the following issues: Proposement Investment Layer

AnaBelgun commented 2 years ago

Pics below to clarify: 1) proposed investment - year based legend for variables in $m image

2) Annual Deferral Value - Constraint season field in info panel no data image

3) Available distribution capacity layer - load and capacity on chart @jego9130 would you please check again the current behaviour in test link vs what is currently available in Production in NationalMap? they seem the same.

NationalMap (previous version of Annual Deferral Value charting image

Test Aug 2022 version of Annual Deferral Value charting image

elangham commented 2 years ago

Thanks @AnaBelgun and @nf-s , almost there!

On the Proposed Investment issue Sorry this is an interpretation issue at our end. However, there are 3 questions raised from looking at this closely:

On Annual deferral layer constraint season issue:

On the Available Distribution capacity legend issue:

I'll keep an eye out if any of these need further responses. Thanks!

jego9130 commented 2 years ago

Hi @AnaBelgun and @nf-s Just touching base on this, do you have any updates? Please let us know if there is anything we need to check or if there is anything we can do to assist you Thanks

AnaBelgun commented 2 years ago

Hi @jego9130 unfortunately we don't have updates at this stage. We'll let you know as soon as we can.

AnaBelgun commented 2 years ago

Dependencies:

nf-s commented 2 years ago

On the Proposed Investment issue

At the moment it is manual because I recreated the existing styling (with the old legend image). See next response

You can use the same color spectrum. It is only manual because I was recreating that old style.

I can make it automatic, but I have to specify the number of legend "ticks" so you get one color per year. You will also lose the "Point size relates to scale of investment" legend item.

For example - here I have specified ticks = 11. (new is on the left, current is the right)

image

I can change it to use this, but if another year were to be added later - then ticks would need to be updated

I'm assuming you mean the "Peak Day Available Capacity" layer - done

On Annual deferral layer constraint season issue:

This is coming from the deferral_value_asset_chain property attached to regions. Here you can see that constraint_season is null.

image

On the Available Distribution capacity legend issue:

I have changed the title to "Forecast load (MVA)"

elangham commented 2 years ago

FAB, THANKS NICK – A FEW RESPONSES IN CAPS BELOW AS I WASN'T SURE HOW THIS WOULD APPEAR AND IF YOU COULD TELL WHICH ONE WAS ME (I'M NOT SHOUTING :P)

On the Proposed Investment issue

* [ ]  Do the legend year bounds need to be updated in the code (e.g. 2023-2031) or is the legend auto generated based on the data?

At the moment it is manual because I recreated the existing styling (with the old legend image). See next response

* [ ]  To have 1 legend colour per year (rather than some having 2 yrs/colour) would require a different colour spectrum, correct? If so, we can live with the current arrangement.

You can use the same color spectrum. It is only manual because I was recreating that old style. I can make it automatic, but I have to specify the number of legend "ticks" so you get one color per year. You will also lose the "Point size relates to scale of investment" legend item. For example - here I have specified ticks = 11. (new is on the left, current is the right)

image

I can change it to use this, but if another year were to be added later - then ticks would need to be updated

I THINK THE DOT SIZE NOTE IS WORTH KEEPING – THE IDEAL OUTCOME WOULD PROBABLY BE THE EXISTING LEGEND BUT USING 10 TICKS (2022-2031) AND THE FINAL ONE BEING "2031 OR LATER" SO IT WOULD CATCH IF THERE ARE A COUPLE OF 2032 ANOMALIES. THE DATA WON'T CHANGE UNTIL AN ANNUAL UPDATE SO NO PROBLEM HAVING A MANUAL SETUP. 

* [x]  Can we please set the default to 6pm on the timeline?

I'm assuming you mean the "Peak Day Available Capacity" layer - done

SORRY, THAT'S WHAT I MEANT - THANKS!

On Annual deferral layer constraint season issue:

* [ ]  If you click 'Show raw data' you can see constraint_season has data but it's just not getting to the Curated view. Great if there's a simple moustache fix. If not, we can live with it.

This is coming from the deferral_value_asset_chain property attached to regions. Here you can see that constraint_season is null.

image

CAN WE CHANGE IT TO PULL FROM CONSTRAINT_SEASON INSTEAD OF deferral_value_asset_chain? I THINK THE FORMER IS CORRECT. LET ME KNOW IF I'M MISCONSTRUING SOMETHING HERE THOUGH

* [x]  Please set the starting year to 2027 on the timeline.

On the Available Distribution capacity legend issue:

* [x]  Yes the current maps don't do this either, but it _should_ show both – I can't recall the conversation with the platform upgrade last year: is this a general issue that 2 series can't be graphed together, or just that it's not specified correctly? It's a platform limitation then I suggest we just change the graph title to just "Forecast load (MVA)"

I have changed the title to "Forecast load (MVA)"

* [x]  Please set the starting year to 2027 on the timeline.

ALL THOSE OTHERS SOUND GOOD, THANKS :)

nf-s commented 2 years ago

I have changed the "Proposed Investment" legend to

image

As for the constraint_season - you can't pull it from outside deferral_value_asset_chain as the value is effectively being overridden by the "local" value in deferral_value_asset_chain.

I don't think we can do anything about this from our side. To fix it we would need to be given another regions file without constraint_season: null in deferral_value_asset_chain

elangham commented 2 years ago

Ok thanks Nick. We won’t reissue the regions but will flag it as an issue we should resolve in next year’s versions.

I think we’re good to go then – let me know if there’s anything we need to look at or OK based on this conversation

From: Nick Forbes-Smith @.> Date: Monday, 12 September 2022 at 5:50 pm To: TerriaJS/nationalmap @.> Cc: Ed Langham @.>, Mention @.> Subject: Re: [TerriaJS/nationalmap] NOM 2022 Constraint Map Update (Issue #1153)

I have changed the "Proposed Investment" legend to

[image]https://user-images.githubusercontent.com/6187649/189595376-4e89146e-22ef-406c-85d8-a1aa0d819f5b.png

As for the constraint_season - you can't pull it from outside deferral_value_asset_chain as the value is effectively being overridden by the "local" value in deferral_value_asset_chain.

I don't think we can do anything about this from our side. To fix it we would need to be given another regions file without constraint_season: null

— Reply to this email directly, view it on GitHubhttps://github.com/TerriaJS/nationalmap/issues/1153#issuecomment-1243348266, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AIBMP354U3XL3TOHPYCNRGLV53ODVANCNFSM5XXQSLAQ. You are receiving this because you were mentioned.Message ID: @.***>

UTS CRICOS Provider Code: 00099F DISCLAIMER: This email message and any accompanying attachments may contain confidential information. If you are not the intended recipient, do not read, use, disseminate, distribute or copy this message or attachments. If you have received this message in error, please notify the sender immediately and delete this message. Any views expressed in this message are those of the individual sender, except where the sender expressly, and with authority, states them to be the views of the University of Technology Sydney. Before opening any attachments, please check them for viruses and defects. Think. Green. Do. Please consider the environment before printing this email.

elangham commented 2 years ago

Hi @nf-s , do you need us to review anything or are you able to advise on the path to release from here? Thanks, Ed

nf-s commented 2 years ago

Hi @elangham

We are just waiting on a new NationalMap release, it should be done this week. After the release, I will push all of the new changes for NOM catalog items.

Cheers, Nick

nf-s commented 2 years ago

Hi @elangham

I have just pushed new NOM changes to production nationalmap.gov.au Sorry it took so long!

Please let me know if find any issues.

Cheers, Nick

nf-s commented 2 years ago

I'm going to close this ticket - feel free to reopen if issues arise