Feedbacks!

[mffreund]

Hi, and thanks for this tool! Looks very promising! Probably you already know, but downloaded winds are inaccurate compared to the latest NOAA forecast😊 And far from the nearest airport very, very unrealistic. I found a hard boundary between 090/70 and 000/05 wind. Do you think there’s really no way to use a geo grid instead of airports for data collection? This would exponentially increase accuracy. Cheers!

[aloys-c]

Hello, thanks for your feedback !

Could you add pictures please if you have the official data loaded in your PFPX, so I can see the extent of the difference ? And regarding the inaccuracy near the airports, is there a small deviation or is it potentially completely wrong too ?

I am using the latest NOAA data as a geo grid. But the interface with PFPX as FSGRW only accepts a format of winds given for each airport, it's then PFPX that does the interpolation and displays the grid. Maybe the interface with Active Sky would be more accurate, but I dont have samples of the files wx_station_list.txt and current_wx_snapshot.txt that they use, so I can convert my data to that format (If someone has Active Sky and is willing to share these files, please do !). From the name I assume it's also a station based data though.

The official server weather files are encrypted in an unknown format so I can't inject data that way. I assume it would be the only way to actually get grid data directly in PFPX...

[mffreund]

Unfortunately I don't use official PFPX wx, so I can't help for this. But I attached a shapshot of wx2pfpx output and a real wind forecast by Skyvector over north Atlantic. As you can see there's a huge difference in both speed and direction. I swear they're both 3/21 06Z. Of course as airports number increases the transitions get smoother. But around an isolated station or in remote places sometimes you even see wind barbs taking opposite directions, for example over Greenland in upper-left picture 1. This makes no sense. I used Active Sky for a while, but had the same issues of the current wx2pfpx: winds taken at airports, inaccurate values and hard transitions over remote areas. I found and old wx_station_list.txt and current_wx_snapshot.txt files and attached them for you to play with. Have you considered the trick of creating fake locations in your airport file? I bet PFPX does only care of their coordinates, so building a geo grid of "fake airports" to inject more data points may actually work.

wx_station_list.txt current_wx_snapshot.txt

[aloys-c]

Thanks you very much for the files, I have been very looking for these two.

I checked carefully, and there was indeed an error in my code, after correcting it the result is much much better, here are some pictures :

Asia :

Australia :

Europe :

US :

Regarding creating new airports stations, I don't think this is possible because the PFPX data is linked to the airac, so the data seems firstly to be encrypted, and also that could possibly be an issue with every airac update... My airport list has the coordinates only to match with the wind data, PFPX only actually cares about the ICAO code for the FSGRW file and just ignores the ones that aren't in the airac...

However, I saw in the file you provided that there seems to be customs stations with, this time, the possibility to provide the coordinates. This could mean that it would be possible to go the Active Sky way and provide unlimited stations. I will investigate that...

[mffreund]

Happy to help! But I redownloaded the file and it looks it's still the older version and cannot test it. Your snapshots looks very good, though. I get your point about adding more locations: if PFPX compares FSGRW ICAO codes with AIRAC, there's no way there. Let's hope the Active Sky way! Long long time ago I also used REX but they stopped support and after a while GFS changed grib structure, making it useless. If I find somewhere an old metar_report.xml file I'll tell you, because in my opinion REX was the highest-quality external weather engine for PFPX.

[aloys-c]

Yes the new version is not uploaded yet, I have been working on the Active Sky Api file and it works really good; it is indeed possible to add an infinite amount of points in addition to the airports. They didn't use that possibility though, and I guess it is because it's quite ressource consuming and not always necessery. When doing interpolation, you just need to get the right compromise because after a certain amount, adding more points improves almost nothing anymore quality-wise, but costs machine time. The way they set their stations is probably the most optimized, looking at how they were spread on the map, looks like they cleary know what they were doing :

I still added a full grid option (65160 points !), because I find it fun and that's cleary in the spirit of what I wanted to do at the begginning. I will release the new version in a few hours..

[aloys-c]

Here it is : wx2pfpx2.0.zip !

Please tell me if you test it and if it works fine on our side too so I can upload it officially 😄

[mffreund]

Hi, I had it a try and I can see big steps forward. Winds aloft close to airport stations are now almost matching my real-world dispatch app (+/- 10kt and +/- 10 degrees maximum deviation). However on the grid version of current_wx_snapshot file I found this anomaly that still causes wrong values on remote areas. Shouldn't they be different grid points with at least some different values?

Another suggestion would be to extend the vertical wind profile up to like FL450 or FL470 as above FL400 it keeps FL400 values. Sadly it seems that PFPX vertical interpolation is even worse than the horizontal one, so it will not be possible to achieve a real world accuracy. But wx2pfpx will make many PFPX users happy!

[mffreund]

Just noticed that the above problem is correlated with grid file on which those dummy stations have the same geo coordinates 😊

[aloys-c]

Thanks for the feedback and with the real values, really encouraging ! Regarding the anomaly a typo on a last minute change and the whole file is indeed nonsense, I just corrected it.

Regarding the flight levels, Active Sky Api for some reason is limited at FL390, I guess the data could be still considered valid and used up to a bit higher, 2000 feets or so as it is half a step between two data levels, so that covers just around the airliners ceilings , but that's for sure not ideal...

wx2pfpx2.1.zip

[mffreund]

Winds at FL390+ have almost constant direction but speed can vary a lot. Executive jets or B787s often use those levels, it's a shame they're not accurate but if it's a limit of the API, there's no way. Btw, while trying to find possible other little inaccuracies, I chose a remote airport in Siberia (UOOO) and verified winds aloft close to it at NOR VOR. Theorically they must match since they're only 0.1 miles away.

Airport winds aloft string is

UOOO::UOOO 221700Z 18005MPS CAVOK M16/M21 Q0999 R19/810250 NOSIG RMK QFE733/0978::*::219,18,-11/237,23,-16/245,23,-21/253,15,-28/320,14,-40/337,17,-52/350,25,-59/337,25,-68/334,23,-71/

So at FL390 334@23kt, - 71°C OAT But looking at PFPX OFP I have

Not a significant difference, but why? Other times this do not occur. For example "O" NDB located AT UOHH airport

UOHH::::::235,4,-18/270,8,-21/277,9,-25/299,11,-33/336,20,-46/349,36,-56/354,40,-60/357,29,-62/9,15,-64/

Here at FL390 both show 009@15, -64°C OAT

Is this just a consequence of PFPX interpolation or a bad interpretation of the additional grid points?

[aloys-c]

These are very small imprecisions for a remote place like that, my guess is it's a mix between interpolation imprecision and also the resolution of the data grid which is 111km. The wind aloft data of an airport is not from the actual airport but it's from the neareast grid point, which is in the worst case 55*sqrt(2) = 78 km away. So that means mapping airports with data brings some shift in the data, which can be an issue when working with also the grid because you have basically the same measureemnt at two different places. I'm trying to see if I can change the grid mode for using the grid and only the grid. Did you notice a real improvement using the grid mode ? Because I'm not sure to what extent the additionnal points are taken into account for the interpolation.... Can I ask also why precision is so important for you ? We have to keep in mind that the NOAA is not real weather but a prediction model with some errors too, when computing an estimated flight performance I don't think these imprecisions change values a lot unless it's really in the wrong direction and speed...

[aloys-c]

So it is apparently not possible to feed only the METAR and TAF via the airports, so even if the grid is used on the whole map there will still be the airports in addition. At this point I think the airports are introducing imprecisions because their data are less accurate than the grid,

I found a workaround, now I'm not taking the nearest point in the grid for the airport data, but I'm doing a small interpolation with the four nearest neighbours, that way the airport data should now be at the right place with the coordinates and not put "stress" on the PFPX interpolation. Maybe that eases a bit the offset you saw at certain stations ?

wx2pfpx2.2.zip

[mffreund]

As I can see this change is overall smoothing the interpolation, meaning we are losing a bit of accuracy AT airports, but improving it almost the same magnitude nearby. This looks good to me. Unfortunately remote areas are still not really improving. Below you can compare 2.2 version output vs dispatch app over waypoints in remote Siberia.

In mid-atlantic I also found this (clearly a grid point in the middle)

But at this point I don't think there's any further room to improvements excluding additional subgridding. Anyway your tool is for sure the best alternative to PFPX server currently, and I think in a while it will be the only way to have weather.

[aloys-c]

Hey thanks for the feedback and data, do the interpolation should increase the accuray at the airport by giving it more realistc measures, it indeed smothens the local values and make it probably easier for PFPX to interpolate. Here's an exemple around EDDM (airfields in orange) :

No interpolation (just taking neareast point value) :

And with interpolations :

And at this stage, I think you're right that the only way to improve accuracy at airports is to use a higher resolution wind data, which I will try now.

One bad news though, I realized that PFPX is not taking into account the extra points of the grid. So only the offical 500 dummy stations are accepted, but the whole grid is just ignored in the file (which explains why I didn't see my error yesterday with the grid file being wrong, it had no effect on the result...). It means that on wx2pfpx2.2, the stations mode is acutally better than the grid one.... Did you look for that older version of the REX file ? I know the last REX doesn't work anymore but if you find the ones that you used in the past then we can see if it allows better results...

There will always be the limits of how PFPX is sampling the data and approximations they used. I will see what happens with a better grid, but if they just round the values in some way, it will probably not improve anything...

The grid point you found in the Atlantic is not related to my data I think, but it's probably a border effect of PFPX doing the interpolation by square of X size, and here you can see the border of their square that could probably not be resolved well with the nearest one...

[mffreund]

Hi, So, sadly, I was hoping to find that REX file on an old HDD, but folder was empty. The only thing I can do is sharing REX installer if you want to install and then try to disassemble it. Maybe it's a stuping thing, but in programming may be not: I noticed this difference in the generated wx_stations_list file between station mode and grid mode.

Station mode:

Grid mode:

Can the issue of ignored stations be related to their id codes not starting from 0 and/or their coordinates missing decimals? So basically unrecognized input format?

[aloys-c]

So I checked and it was indeed a format error, I didn't expect it since it was comma-separated, but now it works again. It's now also possible to use only the grid without the airport data (just to clarify, all data comes from the grid, my airport data (and additional stations) is just interpolated on my side from the NOAA grid data, that's just less points points to process and they are placed in a way that allows speed-optimized interpolation). The whole grid plus the airports is a bit "overkill" but I added an option to still add it for testing (airports are always included in stations mode). METAR and TAF data is independant and always available.

I also added an option to change the resolution from 1° to 0.5 or even 0.25. Just keep in mind that improving the resolution by 2 means 4 times more memory and time for processing, last one doesn't work on my computer. There is some improvement on the airport data (that is generated on wx2pfpx side, see below) using a smaller grid, I would say mostly on places where wind direction is not homogenous. I don't know if it would change a lot in the grid mode (since airports are not included and PFPX does the interpolation from the given grid), but for the stations mode it for sure provides better starting data for the PFPX interpolation. If giving a smaller grid for grid mode to PFPX improves the resulting interpolation it does on the airport locations, that I don't know yet...

UOOO at 1° resolution :

UOOO at 0.5° resolution: So it doesn't change much at that airport in these conditions, but another one below has quite a chang in wind heading and speed, so I guess it can be worth it as an option..

LSGG at 1° resolution :

LSGG at 0.5° resolution:

Here's a beta version of 2.3 :

wx2pfpx2.3b.zip

[mffreund]

Hi, I performed an extensive test and this the report:

• Drastic increase in data download failures (both data missing and not available errors). No difference between modes. Making a new request succeeds after long times. Here:

• GFS 0.25 also crashes the tool on my high-end PC with 64 GB Ram. But this does not look a memory issue as I saw usage peaks of 2GB maximum (nor CPU's, to be honest). However, even if download succeeds, lower resolution data would certainly crash PFPX at loading them. So sadly I would say you to delete the 0.25 grid mode.

• GFS 0.5 works good and for me it's the best compromise between loading times (quite slow but acceptable) and accuracy.

• GFS 0.1 and stations data are of course OK. I think you can leave both there as some people may appreciate playing with different accuracies and at the same time this means no additional work for you :P

Border effects in PFPX square-grid interpolation still exist, but that's a PFPX issue.

[aloys-c]

Thanks for the feedback !

The download issues are more server related, if you download too much you get restricted and depending on the time you download this can also be an issue if it's just after release, I guess a lot of download happen at this time. "Data not available" is not a bug but a feature, it just means it tried to fetch data, but the source has not published it yet, so it's too early.

So as a compromise I put the stations mode on 0.5 (less points so they should be more accurate) and grid mode on 1 (less important since whole map is covered) by default, so casual users should be happy with that. I changed a bit the interface, so it's straightforward to use, advanced settings are available in a setting file though. An export option is added also if anyone would want to have the whole processed data to import somewhere else.

Did you do some values comparison with you dispatch app with the last one ? Are the values still quite far or is it improving with a higher resolution ?

I think we reached the limits of what is possible to do, looking at those continuous and precise-looking alofts makes me quite happy about the result. Still open to suggestion for features tho, but I think the core process is done now...

[mffreund]

We know PFPX interpolates values on a square grid of unknown side. So, if within this square you have a single airport, you get the EXACT real world values OVER that airport and similar values inside the rest of the square. And this is what seems to cause those unavoidable hard transitions along borders with other squares. If there is more than a single airport within the square, or you use grid mode data, PFPX calculates something like an average of the wind alofts injected inside the square. This means you no longer have the exact values over the station, BUT as PFPX takes now an aveage of values, it reduces the average deviation with real world inside the whole square. Transitions along the borders may still be hard, but not as hard as in station mode or lower resolution grid mode. These are noticeable accuracy improvements for long-range flight planning.

A suggestion is making wx2pfpx to download not simply the latest forecast, but the latest READY forecast. For example, at around 08Z I asked a new download. The tool attempted to download GFS 06Z data, which failed as that forecast is released around 10Z, so PFPX loaded the latest downloaded data. In my case yesterday's 12Z. Here I would appreciate to have a refresh to today's 00Z. Quite old, but not as old.

[aloys-c]

Great that's what full grid mode is for, better overall quality and mostly in remote areas. Stations mode (renamed default network) is also way more quicker and I do like the higher precision at airports that would probably match more with the metar. I think that really allows to get everybody happy with the different possibilities...

Regarding the download, that's indeed what wx2pfpx is programmed to do already. Thing is I cannot know which data is ready before trying it (Or maybe I could, but it's so simpler to just try and see, I can hide that step, but I don't think it's a big issue). wx2pfpx is designed to never reuse data and considers it as a fresh start every time you click on download. If the first try at 06z fails, it does tries the one (6hours earlier) 00z, then if it succeded it took that 00z or the program is interupted. At least that's how I coded it, unless there's a bug I didnt identify...

It always worked like that on my side :

Please tell me if you noticed it working differently. You say the script reused the data of yesterday 12z, how do you know that ? It should either show one "success", one "failed" and then "success", or two "failed" and stop. In that last case the data is indeed not updated PFPX indeed shows what's remains in the output folder. I could empty the output folder when clicking on download to get rid of all older data, but I thought in case of download issue people would like to keep their latest download...

[mffreund]

Your description and your snapshot are how I would expect it to be working. But here the problem happened again while trying to download 0.5 data. Request time 15Z, this is the result (2.3b version)

It stops there. 12Z is not out yet, but no attempt to get the 06Z. And, since I was off my computer for a few hours, in my output folder i still have the result of an old successful download which is now outdated. PFPX simply loads what it finds in the folder. It's good if you leave an older forecast after multiple errors, but with the above issue, the older forecast can become way older.

[aloys-c]

Well this is strange, ngl I checked the code and really can't see how something like that would happen. I did some changes though to simplify how I deal with time. Did you try with the last version in the repository ? (you can find it under the code section, it's updated when I save the project, there's the 2.4 now). If it happens again could you try with antivirus deactivated ? For now that's the only thing I see that would stop the script from going on after a fail attempt...

Here it is too : wx2pfpx2.4.zip

The script doesn't update automatically and I didn't plan to add that feature unless someone really finds it usefull, it's really up to the user to update the data, and check the date in the "current data" section when he needs it to generate a flight plan.

[mffreund]

Hi, In this 2.4 version the download process is working good again. However, it's downloading wrong data: If I keep "Use full grid" unchecked, I get the message "grid resolution is 0.5°" and in the output folder I have a 2MB wx file, which is the lowest resolution station mode. If I check that box, the message is "grid resolution is 1°" and in the folder I have an 8MB file (which is ok for 1°), but actually it isn't "full grid".

Seems like messages are inverted and you accidentally deleted the code for 0.5° data download instead of that regarding station mode :)

[aloys-c]

It's not actually, just checked and it's working just fine. Station mode file is always the same size since the number of stations doesn't change. I didn't delete any code, the resolution is just a number. You can set the resolutions you want in the settings.cfg file for each of the modes....

[mffreund]

After a while understanding how this version works, I see no further bugs or weird stuff. It now does what it is expected to do. For me it's a go for release. Excellent job!

[aloys-c]

Okey thanks for feedback, I have just released it.