search

whitewater-guide / gorge

MIT License
2 stars 3 forks source link

Finland #6

Closed Mkrut closed 4 years ago

Mkrut commented 4 years ago

List of station

Arvo = value Virtaama = discharge = расход

interface description

Single station value

Kayaker contact

Licensing. (CC)

ahalkosaari commented 4 years ago

Hi,

Some more links here:

Virtaama Discharge Q [m3/s] http://rajapinnat.ymparisto.fi/api/Hydrologiarajapinta/1.1/odata/Virtaama

LampoPintavesi Surface Water Temperature T [°C] http://rajapinnat.ymparisto.fi/api/Hydrologiarajapinta/1.1/odata/LampoPintavesi

Vedenkorkeus Water level W [cm] http://rajapinnat.ymparisto.fi/api/Hydrologiarajapinta/1.1/odata/Vedenkorkeus

Water level correction between sites, maybe not needed: VedenkTasoTieto Information required for site-to-site conversion between elevation systems. Adding Tasokorjaus (Level correction) to the water level reading gives the water level at the desired altitude system. If the Tasokorjaus value is 0, the water level value is directly entered in the elevation system indicated by the Tasokoordinaatisto field. http://rajapinnat.ymparisto.fi/api/Hydrologiarajapinta/1.1/odata/VedenkTasoTieto

Mkrut commented 4 years ago

The list of all the stations with names, id's and coordinates in JSON

So it's possible to get the value of the flow/level of a specific gauge by running a request described here . Example where Paikka_Id=894 stands for the place id and Aika for the date and time

ahalkosaari commented 4 years ago

How to know which stations are measuring discharge: http://rajapinnat.ymparisto.fi/api/Hydrologiarajapinta/1.1/odata/Suure :

"Suure_Id":1,"Suurekoodi":"W","Nimi":"Vedenkorkeus","Kuvaus":"Vedenkorkeus","NimiEng":"Water level","KuvausEng":"Water level","Yksikko":"cm","taulunnimi":"Vedenkorkeus","DatataulunNimi":"Vedenkorkeus"
"Suure_Id":2,"Suurekoodi":"Q","Nimi":"Virtaama","Kuvaus":"Virtaama","NimiEng":"Discharge","KuvausEng":"Discharge","Yksikko":"m3/s","taulunnimi":"Virtaama","DatataulunNimi":"Virtaama"

and combine that with http://rajapinnat.ymparisto.fi/api/Hydrologiarajapinta/1.1/odata/Laitteisto :

"Laitteisto_Id":7,"Nimi":"Purkautumisk\u00e4yr\u00e4","SuureKoodit":"Q,R"
"Laitteisto_Id":8,"Nimi":"Raporttilomake","SuureKoodit":"W,Q,I,IB"
"Laitteisto_Id":9,"Nimi":"S\u00e4hk\u00f6posti","SuureKoodit":"W,Q,T"

W is for Water level and Q is for discharge. From Laitteisto you can see which Laitteisto_id values you should limit from Paikka list to get list of locations which have for example Discharge measurement. So for discharge Laitteisto_Id should be 7, 8 or 9 and for water level Laitteisto_Id should be 1,2,3,4,5,6,8,9,19,20,23,24,25,27,28,37,38,39,40, or 41.

If you try to get Discharge (or other type of values) from station which does not support it you will get HTTP 404

doomsower commented 4 years ago

@ahalkosaari , @Mkrut

I played around with query builder a bit, but I still need some help.

The plan is:

  1. Get the list of stations here: http://rajapinnat.ymparisto.fi/api/Hydrologiarajapinta/1.1/odata/Paikka. 1.1. I can paginate using $skip and $top 1.2. I can filter by station type: $filter=Laitteisto_Id eq 7 or Laitteisto_Id eq 8 or Laitteisto_Id eq 9 1.3. I can tell it what columns I need: &$select=KoordLat,KoordLong,Paikka_Id,VesalNimi,Laitteisto_Id"
  2. Get the discharge values from http://rajapinnat.ymparisto.fi/api/Hydrologiarajapinta/1.1/#Virtaama

My problems/questions are:

  1. On step 1.1 the endpoint sometimes gives me errors like The 'KoordPkPohj' property on 'Paikka' could not be set to a 'null' value. You must set this property to a non-null value of type 'System.Int32'. I can work around by having a small window, but with thousands gauges this can be stressfull and perhaps I would hit some request limits.
  2. Just how many (approximately) stations do we need for our app? Obviously not every station is useful for kayakers.
  3. Usually we prefer discharge to water level if we can get both. Are there useful stations/spot that only have water level. Do we need it at all? I ask because with so many Laitteisto_Ids I might hit query string length limit.
  4. On step 1.3 I need to get columns for location, unique station code, station name as displayed in our app and (optional) station webpage URL for humans. So
    • KoordLat,KoordLong are for location, just divide by 100
    • Paikka_Id is unique station code
    • Which fields do I need to get to construct station name? There are multiple "nimis". Usually we construct it like <River nane> - <station name>, but we can use any fields to make it convenient for users and editors
  5. I tried to build Virtaama query for single station but failed. However, Mike's example above works. But how do you set date? Obviously I cannot know exact date to use as query parameter. Usually I just need one latest value.
  6. Can I use miltiple Paikka_Ids to get Virtaama for many stations at once? Ideally I'd like to get all the stations in one query.
  7. What timezone are dates in?
ahalkosaari commented 4 years ago

Here is some answers, hope these help: 1) I get same errors when I get to the end of list (bug?). Example $skip=3150 works but $skip=3200 (or more) gives that error. 2) Rough guess would be between 100 to 150 stations. I will update this later. 3) We prefer discharge. I will check if water level is relevant anywhere. Actually fishermen could use this app to check water levels :) 4) You could use - - :

doomsower commented 4 years ago

Thank you!

You could use - - :

I think you missed something here. What we need here is combination of names and codes that is familiar to user. To put it simple, you'll see these names in a list when you (as an editor) choose section's gauge.

doomsower commented 4 years ago

I get same errors when I get to the end of list (bug?). Example $skip=3150 works but $skip=3200 (or more) gives that error.

Maybe you could report this bug to them?

ahalkosaari commented 4 years ago

Thank you!

You could use - - :

I think you missed something here. What we need here is combination of names and codes that is familiar to user. To put it simple, you'll see these names in a list when you (as an editor) choose section's gauge.

Oops :D some of my answer disappeared.

I meant that user-friendly list would contain: Kuntanimi - Paikka - Nro (Example Kotka-Pernoonkoski-1410410 (because Kuntanimi (city/town name) is more accurate than water area name (which might be really wide area). Nro is same station number which is mentioned at water level web pages. You can also google that number and (from finnish pages) find the web page of station.

Maybe you could report this bug to them?

I will.

doomsower commented 4 years ago

I've added Finland to beta version of our project. Check it out here

As usual, figuring out what to harvest takes longer than actually writing the code. There are some problems.

First, some 7-digit ids do not map to URLs. For example this one. Either me or @ahalkosaari got the URL schema wrong.

Second one I'm not sure about, but maybe we should not harvest some gauges? They look kinda similar, same Nro and Paikka_Id differ by one.

изображение

If you compare two here, the difference is in suure_id. I harvested only discharge gauges and I filtered by Laitteisto_Id eq 7 or Laitteisto_Id eq 8 or Laitteisto_Id eq 9. Maybe I should filter by suure_id eq 2 instead?

ahalkosaari commented 4 years ago

I've added Finland to beta version of our project. Check it out here

Awesome :)

First, some 7-digit ids do not map to URLs. For example this one. Either me or @ahalkosaari got the URL schema wrong. That station (2279 | 1 | 79 | Harjavalta | 3510400 | Harjavalta, ylä) does not have web page at all (and that Paikka_Id has only water level information). Nearest readings are from station 3510450 which has also discharge.

If you compare two here, the difference is in suure_id. I harvested only discharge gauges and I filtered by Laitteisto_Id eq 7 or Laitteisto_Id eq 8 or Laitteisto_Id eq 9. Maybe I should filter by suure_id eq 2 instead?

Yes, that would be more straightforward (Suure_Id tells straight what kind of measurement can be read. Sorry I didn't notice this earlier):

So you have to read Water level from Paikka_Id 2705 and Discharge from Paikka_Id 2706

ahalkosaari commented 4 years ago

Maybe you could report this bug to them? I will.

They replied, and there is some misconfiguration between open data database and production database. It will be fixed.

doomsower commented 4 years ago

That station (2279 | 1 | 79 | Harjavalta | 3510400 | Harjavalta, ylä) does not have web page at all (and that Paikka_Id has only water level information). Nearest readings are from station 3510450 which has also discharge.

Yeah but how do I know whether I should construct gauge URL from Nro or leave it blank?

ahalkosaari commented 4 years ago

That station (2279 | 1 | 79 | Harjavalta | 3510400 | Harjavalta, ylä) does not have web page at all (and that Paikka_Id has only water level information). Nearest readings are from station 3510450 which has also discharge.

Yeah but how do I know whether I should construct gauge URL from Nro or leave it blank?

Could it be admin editable content, which would have default value url generated from Nro? I think that when enabling cauges the admin should check that url works.

doomsower commented 4 years ago

I've updated the script and re-harvested gauges list on beta. Check it out.

Regarding URLs. It's possible to edit them manually, but only admins (not regular region editors) can do that. And I don't plan to change this, because this is not very important feature.

I checked their website, and the URL schema http://wwwi2.ymparisto.fi/i2/<First 2 numbers>/qy/wqfi.html seems to be more complex. There're at least 4 variants: qy, ly, qx, lx. Since the maintainers are responsive, maybe you could ask them about urls? Otherwise let me know if I can deploy this in production.

ahalkosaari commented 4 years ago

Maybe you could report this bug to them? I will.

They replied, and there is some misconfiguration between open data database and production database. It will be fixed.

They informed me that this is now fixed. I tested and don't get the error anymore.

ahalkosaari commented 4 years ago

I checked their website, and the URL schema http://wwwi2.ymparisto.fi/i2/<First 2 numbers>/qy/wqfi.html seems to be more complex. There're at least 4 variants: qy, ly, qx, lx. Since the maintainers are responsive, maybe you could ask them about urls? Otherwise let me know if I can deploy this in production.

I asked about the urls, but I think that we could deploy current setup and see that values are coming in allright. We can always add the correct link manually to section description.

doomsower commented 4 years ago

@ahalkosaari

Finland is live on production. But to begin harvesting, you must connect at least one gauge to one section. I thought I could do this myself, but could not match any of existing sections.

image

I find it weird that stations are kinda clustered. And all the existing sections are in between clusters.

Please ping me here after you link any gauge. Also please do not confuse beta and production version.

ahalkosaari commented 4 years ago

I linked one gauge. Should I make it visible to users?

doomsower commented 4 years ago

I've enabled data harvesting. FYI, it only harvests those gauges that have sections connected (otherwise it will flood our database very quickly)

ahalkosaari commented 4 years ago

Is there any recommendation how to link multiple cauges to one section? For example one level and one discharge?

doomsower commented 4 years ago

Is there any recommendation how to link multiple cauges to one section? For example one level and one discharge?

This is not possible now. In our system, a single gauge can produce both flow and level values. We try to provide both where upstream data sources provide both. But this is not very important to us, sometimes we skip levels and harvest discharge only for the sake of technical simplicity.

Here in Finland I see no way to merge two Paikkas into one gauge. They do not share a single reliable identifier.

Mkrut commented 4 years ago

I linked one gauge. Should I make it visible to users?

Yes you can make it visible. The region is hidden yet. Meaning only you and us can see the sections. Let's publish it when you link all the gauges you want

doomsower commented 4 years ago

image

@ahalkosaari I think I screwed up the coordinate system initially. I fixed it on beta. Compare this screenshot with screenshot from production above. Did I get it right this time? Please approve

As a consequence, we'll have to delete and re-harvest gauges on production, and re-bind them.

ahalkosaari commented 4 years ago

@ahalkosaari I think I screwed up the coordinate system initially. I fixed it on beta. Compare this screenshot with screenshot from production above. Did I get it right this time? Please approve

As a consequence, we'll have to delete and re-harvest gauges on production, and re-bind them.

This looks better.

Btw. I made a request to hydrology information provider about data availability per date. It seems that Odata api does not give out readings for current (ongoing) day, only yesterday (current day gives http 404). However on hydrology data station pages there is data for also current day.

ahalkosaari commented 4 years ago

Btw. I made a request to hydrology information provider about data availability per date. It seems that Odata api does not give out readings for current (ongoing) day, only yesterday (current day gives http 404). However on hydrology data station pages there is data for also current day.

They replied that this is a "feature". Open data is copied from production data backup from earlier day so there is one day delay.

doomsower commented 4 years ago

I have re-harvested gauges on production. The locations are now correct. But you have to re-bind most of sections. Sorry about that.

Mkrut commented 4 years ago

@ahalkosaari should we open the region for public yet? I thinks it's a good idea to do it now while there is still some water in the country

ahalkosaari commented 4 years ago

@ahalkosaari should we open the region for public yet? I thinks it's a good idea to do it now while there is still some water in the country

:D Yes, it would be good to do now. Section descriptions etc are not ready yet but at least there is some data.

Btw. I noticed that in some point the hydrology data got freshier for some sections (not 2 days old anymore - values were actually from the same day).

Mkrut commented 4 years ago

Cool! May I ask you to search a nice horizontal picture to put it on the cover please? Minimum 2048 pixels wide. Btw is it fine if I tag you in the post announcing Finland? What's your instagram?