wx2pfpx v2.7

[PinkPony89]

Hello again Aloys,

I submitted a new pull request for v2.7 As you mentioned my previous release was getting stuck on "Downloading last weather reports..." I tested it all day and it was very inconsistent. Sometime it would work sometimes it wouldn't, which make me look closer at the new Dataserver URL under the new API.

I noticed that the METAR .xml output was identical between the old https://aviationweather-cprk.ncep.noaa.gov/adds/dataserver_current/httpparam? and the new new https://aviationweather.gov/cgi-bin/data/dataserver.php? very unlikely that this was causing the download to get stuck.

However as I eventually realized the TAF outputs are indeed different!

old using EGLL as an example

https://aviationweather-cprk.ncep.noaa.gov/adds/dataserver_current/httpparam?datasource=tafs&requestType=retrieve&fields=raw_text,station_id&format=xml&mostRecentForEachStation=constraint&hoursBeforeNow=3&stationString=EGLL

<response xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" version="1.2" xsi:noNamespaceSchemaLocation="http://www.aviationweather.gov/static/adds/schema/taf1_2.xsd">
<request_index>214706</request_index>
<data_source name="tafs"/>
<request type="retrieve"/>
<errors/>
<warnings/>
<time_taken_ms>3</time_taken_ms>
<data num_results="1">
<TAF>
<raw_text>TAF EGLL 262300Z 2700/2806 20004KT 9999 SCT030 TEMPO 2700/2709 8000 PROB30 TEMPO 2700/2708 0300 FG VV/// TEMPO 2704/2710 BKN005 PROB30 TEMPO 2709/2716 6000 SHRA BKN012</raw_text>
<station_id>EGLL</station_id>
<forecast>
<sky_condition/>
</forecast>
<forecast/>
<forecast/>
<forecast>
<sky_condition/>
</forecast>
<forecast>
<sky_condition/>
</forecast>
</TAF>
</data>
</response>

new using EGLL as an example

https://aviationweather.gov/cgi-bin/data/dataserver.php?datasource=tafs&requestType=retrieve&fields=raw_text,station_id&format=xml&mostRecentForEachStation=constraint&hoursBeforeNow=3&stationString=EGLL

<response xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" version="1.3" xsi:noNamespaceSchemaLocation="https://aviationweather.gov/data/schema/taf1_3.xsd">
<request_index>1698382661</request_index>
<data_source name="tafs"/>
<request type="retrieve"/>
<errors/>
<warnings/>
<time_taken_ms>135</time_taken_ms>
<data num_results="1">
<TAF>
<raw_text>TAF EGLL 262300Z 2700/2806 20004KT 9999 SCT030 TEMPO 2700/2709 8000 PROB30 TEMPO 2700/2708 0300 FG VV/// TEMPO 2704/2710 BKN005 PROB30 TEMPO 2709/2716 6000 SHRA BKN012</raw_text>
<station_id>EGLL</station_id>
<forecast>
<fcst_time_from>2023-10-27T00:00:00Z</fcst_time_from>
<fcst_time_to>2023-10-28T06:00:00Z</fcst_time_to>
<wind_dir_degrees>200</wind_dir_degrees>
<wind_speed_kt>4</wind_speed_kt>
<visibility_statute_mi>6+</visibility_statute_mi>
<sky_condition sky_cover="SCT" cloud_base_ft_agl="3000"/>
</forecast>
<forecast>
<fcst_time_from>2023-10-27T00:00:00Z</fcst_time_from>
<fcst_time_to>2023-10-27T09:00:00Z</fcst_time_to>
<change_indicator>TEMPO</change_indicator>
<visibility_statute_mi>4.97</visibility_statute_mi>
</forecast>
<forecast>
<fcst_time_from>2023-10-27T00:00:00Z</fcst_time_from>
<fcst_time_to>2023-10-27T08:00:00Z</fcst_time_to>
<change_indicator>TEMPO</change_indicator>
<probability>30</probability>
<visibility_statute_mi>0.19</visibility_statute_mi>
<wx_string>FG</wx_string>
</forecast>
<forecast>
<fcst_time_from>2023-10-27T04:00:00Z</fcst_time_from>
<fcst_time_to>2023-10-27T10:00:00Z</fcst_time_to>
<change_indicator>TEMPO</change_indicator>
<sky_condition sky_cover="BKN" cloud_base_ft_agl="500"/>
</forecast>
<forecast>
<fcst_time_from>2023-10-27T09:00:00Z</fcst_time_from>
<fcst_time_to>2023-10-27T16:00:00Z</fcst_time_to>
<change_indicator>TEMPO</change_indicator>
<probability>30</probability>
<visibility_statute_mi>3.73</visibility_statute_mi>
<wx_string>SHRA</wx_string>
<sky_condition sky_cover="BKN" cloud_base_ft_agl="1200"/>
</forecast>
</TAF>
</data>
</response>

My suspicion is that the additional lines in the API Dataserver .xml output when using the new https://aviationweather.gov/cgi-bin/data/dataserver.php? TAF URL are somehow causing a problem during processing of the TAF. Maybe before/during to the xml2array process?

I have updated the METAR in v2.7 to the new API Dataserver URL. However I am still using the old fallback TAF URL https://aviationweather-cprk.ncep.noaa.gov/ as using the new TAF Dataserver URL is causing inconsistent downloads.

I made ~100 download attempts ranging from 1 to 6 forecasts and all of them were successful. v2.7 is now successful at downloading METAR/TAF each time. Please let me know if you are experienced the same?

I will continue to investigate what changes in the wx2pfpx code may be necessary to process the new TAF code consistently.

I still suspect that https://aviationweather-cprk.ncep.noaa.gov/ will eventually be "retired" permanently once NOAA and the stakeholders are satisfied with the new API.

The new TAF Dataserver URL is https://aviationweather.gov/cgi-bin/data/dataserver.php?datasource=tafs&requestType=retrieve&fields=raw_text,station_id&format=xml&mostRecentForEachStation=constraint&hoursBeforeNow=3&stationString=

If you have any time to check this, maybe you can determine what changes in the wx2pfpx code are necessary to consistently process download/process the new .xml output when using the new TAF Dataserver URL

Leo

[aloys-c]

Hi Leo,

You are probably right the function xml2array was quite unstable and depending on very strict structure to work, I think that what should need to be updated. Since the v2.7 doesn't add anything to the v.2.6 I think we should keep the latter for the moment and migrate to 2.7 once the code is fully working with both TAFS and METARS on new url...

By the way, I don't think the code in the "utils" folder is really called regarding the parsing of the stations, this was old code that is kept just in case...

Aloys