Bug: Download URL's fail in particular Australia BOM

[rgleason]

The fail with the same error as

Export to kap corrupted. https://github.com/rgleason/weatherfax_pi/issues/53

We think it is due to methods used in Weatherfax.

[rgleason]

BOM appears to flag automated web requests and prevent their use. Suggested by Oziolli as a possible cause. CF Post #1501

BOM FTP server - They may charge for services http://www.bom.gov.au/catalogue/anon-ftp.shtml

[rgleason]

From this page http://www.bom.gov.au/catalogue/anon-ftp.shtml I first picked "Satellites" and was asked what program to use for FTP I had installed filezilla and so picked that. Then some configuring was required. Using Filezilla FTP Client and FTP Protocol Host: bom.gov.au Encryption: Plain FTP insecure Logon type: Ask for Password User: "enter something' Password: "nothing"

remote site: /anon/gen/gms Initial connection using the Satellite link, then showed a number of file names. Selection of one to download did succeed, it was a tif and does display.

[rgleason]

From this page http://www.bom.gov.au/catalogue/anon-ftp.shtml I first picked "Model Chart Images" and was asked what program to use for FTP I picked filezila remote site: /anon/gen/charts_data

Filenames appeared. I then downloaded several *.tar I had difficulty opening them. They may be encrypted or something.

[rgleason]

From this page http://www.bom.gov.au/catalogue/anon-ftp.shtml I first picked "Low Resolution Chart Products" and was asked what program to use for FTP I picked filezila remote site: /anon/gen/difacs

Filenames appeared. I then downloaded 18 files and can open them. They are not interpreted, but they do provide data.

[rgleason]

Pick "Media Graphics Products"
remote directory: /anon/gen/nmgu is not so useful

[rgleason]

Since files are available using ftp, I wonder if ftp can be used in weatherfax_pi to download. Then it is a matter locating and using the right files.

It would be worthwhile to develop the list of files that should be used, while we figure out if ftp can be used in the plugin

[ozolli]

I managed to download the right image with lftp on linux: CoreELEC:~ # lftp ftp.bom.gov.au lftp ftp.bom.gov.au:/> get anon/gen/fwo/IDY00030.202205070600.png 184433 bytes transferred in 5 seconds (39.5 KiB/s)
lftp ftp.bom.gov.au:/> exit

[rgleason]

Copyright at the bottom of the page: http://www.bom.gov.au/catalogue/anon-ftp.shtml All products available via the anonymous FTP service are subject to the default terms of the Bureau's copyright notice: you may download, use and copy that material for personal use, or use within your organization but you may not supply that material to any other person or use it for any commercial purpose. Users intending to publish Bureau data should do so as Registered Users.

Are we outside the copyright? .being opensource? ..are OpenCPN users part of the OpenCPN organization?

Using the FTP Service http://www.bom.gov.au/catalogue/anon-ftp-hints.shtml

[ozolli]

No we are not outside the copyright. OpenCPN provides no BOM resource to the end user, it provides a way to get those resources for their own personal use.

[rgleason]

Good. I agree.

Now we need to get two things done.

1. Build a small and useful current list of ftp url and filenames that we want to use for BOM.
2. Find someone to help weatherfax use ftp instead of curl or wget to download.

[rgleason]

Did not intend to close

From the http server http://www.bom.gov.au/fwo/IDY00030.202205071800.png

From the ftp server using ftp filezilla ftp.bom.gov.au/ Connecting to 134.178.253.145:21... Retrieving directory listing of "/anon/gen/difacs"...

For example IDX0078.202205061800.gif MSLP Analysis Access Global Valid UTC 1800 Friday May 06

IDX0078 - Type of File/Data - We need a list of the Types that are important. 2022 Year - This yearly value is typical, no further info required 05 Month - This monthly value is typical, no further info required. 06 Day - This daily value is typical, no further info required. 1800 Valid UTC time - Need the typical "Valid UTC" Times that are used in the cycle for this file type as they vary.

The files of this type IDX0078 that we find in this ftp directory are IDX0078.202205060600.gif
IDX0078.202205061200.gif IDX0078.202205061800.gif IDX0078.202205070000.gif IDX0078.202205070600.gif They are all valid as of the time indicated above. They all say "Contour Interval 4" and they are Issued at different times that are noted at the bottom

From WeatherfaxInternetRetrieval_misc.xml file Examples that use StrDate "string-date" It is reasonably clear how this function is used.

  <Server Name="Weather Online" Url="http://www.weatheronline.co.uk/daten/sailcharts/gfs/">
    <Region Name="Great Britain">
      <Map Url="%Y/%m/%d/pslv_grbr_06-%Y%m%d%H.gif" Area="1" Hour="-9;6;0" />
      <Area Name="1" lat1="48N" lat2="60N" lon1="12W" lon2="6E" />
    </Region>
    <Region Name="Asia">
      <Map Url="%Y/%m/%d/pslv_asie_06-%Y%m%d%H.gif" Contents="Surface Pressure (GFS)" Area="1" Hour="-9;6;0" />
      <Map Url="gfs/%Y/%m/%d/pslv_asie_00-%Y%m%d%H.gif" Contents="Surface Pressure (GFS)" Area="1" Hour="-15;6;0" />
      <!-- many more -->
      <Area Name="1" lat1="10S" lat2="45N" lon1="50E" lon2="150E" />
    </Region>
  </Server>
  <Server Name="Singapore Met" Url="http://www.weather.gov.sg/files/marine/">
    <Region Name="Singapore">
      <Map Url="surfacewinds/surfacewinds_24Hr_%Y%m%d_213000.gif" Contents="Surface Wind Forecast 24hr" Area="1" />
      <Map Url="surfacewinds/surfacewinds_48Hr_%Y%m%d_213000.gif" Contents="Surface Wind Forecast 48hr" Area="1" />
      <Map Url="surfacewinds/surfacewinds_72Hr_%Y%m%d_213000.gif" Contents="Surface Wind Forecast 72hr" Area="1" />
      <Map Url="waveheight/sig_waves_height_24Hr_%Y%m%d_065000.gif" Contents="Significant Wave Forecast 24hr" Area="2" />
      <Map Url="waveheight/sig_waves_height_48Hr_%Y%m%d_065000.gif" Contents="Significant Wave Forecast 48hr" Area="2" />
      <Map Url="waveheight/sig_waves_height_72Hr_%Y%m%d_065000.gif" Contents="Significant Wave Forecast 72hr" Area="2" />
      <Map Url="swell/swell_24Hr_%Y%m%d_100000.gif" Contents="Swell Wave Ht 24hr" Area="3" />
      <Map Url="swell/swell_48Hr_%Y%m%d_100000.gif" Contents="Swell Wave Ht 48hr" Area="3" />
      <Map Url="swell/swell_72Hr_%Y%m%d_100000.gif" Contents="Swell Wave Ht 72hr" Area="3" />
      <Area Name="1" lat1="20S" lat2="40N" lon1="80E" lon2="150E" />
      <Area Name="2" lat1="15S" lat2="35N" lon1="90E" lon2="130E" />
      <Area Name="3" lat1="20S" lat2="40N" lon1="80E" lon2="150E" />
    </Region>
  </Server>

[ozolli]

I have tried to modify WeatherFaxInternetRetrieval_Australia.xml to point to the ftp URLs. The result is a corrupted file as the plugin is using the API function OCPN_downloadFile. This function uses curl to download a file but it sets it as an ASCII file, thus the corrupted result. This is possible to use curl to download a binary file using the --output parameter but unfortunately OCPN_downloadFile function has no such parameter...

[ozolli]

I then downloaded several *.tar I had difficulty opening them. They may be encrypted or something.

I downloaded one tar and could open it fine with linux. Inside is the directory hierarchy you could read in WeatherFaxInternetRetrieval_Australia.xml (like /windbarb/10m/)

[rgleason]

IDX0405 MSLP / 1000-500hpa thickness Access Global 24hr Forecast Date and Time (typical for this file/data type) IDX0405.202205070000 IDX0405.202205070600 IDX0405.202205071200 IDX0405.202205071800

[rgleason]

@ozolli It might be easiest if we can use the *.tar files! Perhaps I am not expanding it with the right tools, I was using 7z.

Then all we have to get solved is get weatherfax to download using ftp.

[rgleason]

ozolli wrote:

I have tried to modify WeatherFaxInternetRetrieval_Australia.xml to point to the ftp URLs. The result is a corrupted file as the plugin is using the API function OCPN_downloadFile. This function uses curl to download a file but it sets it as an ASCII file, thus the corrupted result.
This is possible to use curl to download a binary file using the --output parameter but unfortunately OCPN_downloadFile function has no such parameter...

Thank you. I was going to try that. This is very useful. I'll ask about it.

[rgleason]

Besides that, these are the best weatherfax charts we've found, ones like this one and they have fronts and are annotated! http://www.bom.gov.au/fwo/IDY00030.202205071800.pn g

[rgleason]

I've posted this in Opencpn/Plugins Issues on github. https://github.com/OpenCPN/plugins/issues/674

@Ozolli Please add anything you think needs to be said.

[rgleason]

I am able to ftp access and download bom.gov.au /anon/gen/fwo there are many different file types, html, txt, xml, gif, png, etc.

[ozolli]

@ozolli It might be easiest if we can use the *.tar files!

Yes but the tar files are circa 250MB...

I have commented the issue in Opencpn/Plugins.

[rgleason]

File Search for Download in the src directory

Search "download" (12 hits in 3 files of 155 searched)
  C:\Users\fcgle\source\weatherfax_pi\src\InternetRetrievalDialog.cpp (5 hits)
    Line 532: /* put download in a thread because the wx routines are all blocking
    Line 535: class DownloadThread : public wxThread
    Line 538:     DownloadThread(wxString fu, wxString fn)
    Line 648:         _OCPN_DLStatus res = OCPN_downloadFile( url, filename, _("WeatherFax InternetRetrieval"),
    Line 663:             wxMessageDialog mdlg(this, _("Failed to Download: ") +
  C:\Users\fcgle\source\weatherfax_pi\src\WeatherFax.cpp (6 hits)
    Line 789: bool WeatherFax::DownloadFile( wxString filename )
    Line 794:     _OCPN_DLStatus res = OCPN_downloadFile( url, tmp_filename, _("WeatherFax Data update"),
    Line 813:             wxMessageDialog mdlg(this, _("Failed to Download: ") +
    Line 830:     if( DownloadFile( _T("WeatherFaxInternetRetrieval.xml") ) &&
    Line 831:        DownloadFile( _T("CoordinateSets.xml") ) ) {
    Line 834:     if( DownloadFile( _T("WeatherFaxSchedules.xml") ) ) {
  C:\Users\fcgle\source\weatherfax_pi\src\WeatherFax.h (1 hit)
    Line 128:     bool DownloadFile( wxString filename );

[rgleason]

File search for OCPN_downloadFile on entire weatherfax code, then select items

  C:\Users\fcgle\source\weatherfax_pi\libs\ocpn-api\ocpn_plugin.h (2 hits)
    Line 1184: extern DECL_EXP _OCPN_DLStatus OCPN_downloadFile( const wxString& url, const wxString &outputFile, 
    Line 1192: extern DECL_EXP _OCPN_DLStatus OCPN_downloadFileBackground( const wxString& url, const wxString &outputFile,
  C:\Users\fcgle\source\weatherfax_pi\src\InternetRetrievalDialog.cpp (1 hit)
    Line 648:         _OCPN_DLStatus res = OCPN_downloadFile( url, filename, _("WeatherFax InternetRetrieval"),
  C:\Users\fcgle\source\weatherfax_pi\src\WeatherFax.cpp (1 hit)
    Line 794:     _OCPN_DLStatus res = OCPN_downloadFile( url, tmp_filename, _("WeatherFax Data update"),

[rgleason]

From ocpn_plugin.h

/*   Synchronous (Blocking) download of a single file  */

extern DECL_EXP _OCPN_DLStatus OCPN_downloadFile( const wxString& url, const wxString &outputFile, 
                                       const wxString &title, const wxString &message, 
                                       const wxBitmap& bitmap,
                                       wxWindow *parent, long style, int timeout_secs);

/*   Asynchronous (Background) download of a single file  */

extern DECL_EXP _OCPN_DLStatus OCPN_downloadFileBackground( const wxString& url, const wxString &outputFile,
                                                  wxEvtHandler *handler, long *handle);

extern DECL_EXP void OCPN_cancelDownloadFileBackground( long handle );

/*   Synchronous (Blocking) HTTP POST operation for small amounts of data */

extern DECL_EXP _OCPN_DLStatus OCPN_postDataHttp( const wxString& url, const wxString& parameters, wxString& result, int timeout_secs );

/*   Check whether connection to the Internet is working */

[rgleason]

Fixed by bdbcat with ftp. Thank you!

[rgleason]

Links reviewed and fixed by Ozolli. Thank you!