search

ggebbie / TMI.jl

Total Matrix Intercomparison for Julia
MIT License
8 stars 3 forks source link

NetCDF: Unknown file format #148

Open ggebbie opened 9 months ago

ggebbie commented 9 months ago 
A
nordic: Error During Test at /home/runner/work/TMI.jl/TMI.jl/test/test_nordic.jl:1
  Got exception outside of a @test
  NetCDF error code -51:
    NetCDF: Unknown file format
  Stacktrace:
    [1] check
      @ ~/.julia/packages/NetCDF/7hOe9/src/netcdf_helpers.jl:22 [inlined]
    [2] nc_open(ath::String, mode::UInt16, cidp::Vector{Int32})
      @ NetCDF ~/.julia/packages/NetCDF/7hOe9/src/netcdf_c.jl:55
    [3] nc_open
      @ ~/.julia/packages/NetCDF/7hOe9/src/netcdf_helpers.jl:103 [inlined]
    [4] open(fil::String; mode::UInt16, readdimvar::Bool, add_finalizer::Bool)
      @ NetCDF ~/.julia/packages/NetCDF/7hOe9/src/NetCDF.jl:913
    [5] open
      @ ~/.julia/packages/NetCDF/7hOe9/src/NetCDF.jl:910 [inlined]
    [6] open(f::NetCDF.var"#55#56"{String}, args::String; kwargs::@Kwargs{})
      @ NetCDF ~/.julia/packages/NetCDF/7hOe9/src/NetCDF.jl:999
    [7] open
      @ ~/.julia/packages/NetCDF/7hOe9/src/NetCDF.jl:998 [inlined]
    [8] #ncread#54
      @ ~/.julia/packages/NetCDF/7hOe9/src/NetCDF.jl:1052 [inlined]
    [9] ncread
      @ ~/.julia/packages/NetCDF/7hOe9/src/NetCDF.jl:1046 [inlined]
   [10] watermassmatrix(file::String)
      @ TMI ~/work/TMI.jl/TMI.jl/src/config.jl:272
ggebbie commented 8 months ago

Both google drive and onedrive do not work for large files because of virus scanner. Need alternative hosting method.

b-r-hamilton commented 2 months ago

Hi Jake - I'm getting this issue again for the 2deg x 2deg file. I just double-checked by downloading a fresh copy of TMI, and then running config_from_nc(versionlist()[2]) and I get the following error

julia> config_from_nc(versionlist()[2])
workaround for 2° x 2°
--2024-09-04 12:42:41--  https://docs.google.com/uc?export=download&confirm=&id=1-YEkB_YeQGqPRH6kauhBb2bi_BjVGt9b
Resolving docs.google.com (docs.google.com)... 142.250.65.174, 2607:f8b0:4006:80e::200e
Connecting to docs.google.com (docs.google.com)|142.250.65.174|:443... connected.
HTTP request sent, awaiting response... 303 See Other
Location: https://drive.usercontent.google.com/download?id=1-YEkB_YeQGqPRH6kauhBb2bi_BjVGt9b&export=download [following]
--2024-09-04 12:42:41--  https://drive.usercontent.google.com/download?id=1-YEkB_YeQGqPRH6kauhBb2bi_BjVGt9b&export=download
Resolving drive.usercontent.google.com (drive.usercontent.google.com)... 142.250.80.1, 2607:f8b0:4006:80a::2001
Connecting to drive.usercontent.google.com (drive.usercontent.google.com)|142.250.80.1|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2445 (2.4K) [text/html]
Saving to: ‘TMI_modern_180x90x33_GH11_GH12.nc’

TMI_modern_180x90x3 100%[===================>]   2.39K  --.-KB/s    in 0s      

2024-09-04 12:42:41 (24.3 MB/s) - ‘TMI_modern_180x90x33_GH11_GH12.nc’ saved [2445/2445]

A
ERROR: NetCDF error code -51:
    NetCDF: Unknown file format
Stacktrace:
  [1] check
    @ ~/.julia/packages/NetCDF/49usH/src/netcdf_helpers.jl:22 [inlined]
  [2] nc_open(ath::String, mode::UInt16, cidp::Vector{Int32})
    @ NetCDF ~/.julia/packages/NetCDF/49usH/src/netcdf_c.jl:55
  [3] nc_open
    @ ~/.julia/packages/NetCDF/49usH/src/netcdf_helpers.jl:103 [inlined]
  [4] open(fil::String; mode::UInt16, readdimvar::Bool, add_finalizer::Bool)
    @ NetCDF ~/.julia/packages/NetCDF/49usH/src/NetCDF.jl:912
  [5] open
    @ ~/.julia/packages/NetCDF/49usH/src/NetCDF.jl:909 [inlined]
  [6] open(f::NetCDF.var"#55#56"{String}, args::String; kwargs::@Kwargs{})
    @ NetCDF ~/.julia/packages/NetCDF/49usH/src/NetCDF.jl:998
  [7] open
    @ ~/.julia/packages/NetCDF/49usH/src/NetCDF.jl:997 [inlined]
  [8] #ncread#54
    @ ~/.julia/packages/NetCDF/49usH/src/NetCDF.jl:1051 [inlined]
  [9] ncread
    @ ~/.julia/packages/NetCDF/49usH/src/NetCDF.jl:1045 [inlined]
 [10] watermassmatrix(file::String)
    @ TMI ~/Downloads/TMI.jl/src/config.jl:275
 [11] macro expansion
    @ ./timing.jl:279 [inlined]
 [12] config_from_nc(TMIversion::String; compute_lu::Bool)
    @ TMI ~/Downloads/TMI.jl/src/config.jl:18
 [13] config_from_nc(TMIversion::String)
    @ TMI ~/Downloads/TMI.jl/src/config.jl:13
 [14] top-level scope
    @ REPL[11]:1

Not sure if the file is corrupted or something!

b-r-hamilton commented 2 months ago

I looped through all versions

julia> for v in versionlist()
       try 
       config_from_nc(v)
       catch
       println("error with " * v) 
       end
       end

and also found that the LGM_90x45x33_G14A throws an error

julia> config_from_nc("LGM_90x45x33_G14A")
read via GoogleDrive.jl
┌ Info: Downloading
│   source = "https://drive.usercontent.google.com/download?id=1ADkDI3Fc3z4Vm75K5u6hx0Yu1P0iVnW1&export=download"
│   dest = "/home/brynn/Downloads/TMI.jl/data/download"
│   progress = 1.0
│   time_taken = "0.0 s"
│   time_remaining = "0.0 s"
│   average_speed = "1.566 MiB/s"
│   downloaded = "1.604 KiB"
│   remaining = "0 bytes"
└   total = "1.604 KiB"
ERROR: HTTP.Exceptions.StatusError(404, "GET", "/download?id=1ADkDI3Fc3z4Vm75K5u6hx0Yu1P0iVnW1&export=download", HTTP.Messages.Response:
"""
HTTP/1.1 404 Not Found
Content-Type: text/html; charset=utf-8
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Pragma: no-cache
Expires: Mon, 01 Jan 1990 00:00:00 GMT
Date: Wed, 04 Sep 2024 16:50:16 GMT
Accept-CH: Sec-CH-UA-Arch, Sec-CH-UA-Bitness, Sec-CH-UA-Full-Version, Sec-CH-UA-Full-Version-List, Sec-CH-UA-Model, Sec-CH-UA-WoW64, Sec-CH-UA-Form-Factors, Sec-CH-UA-Platform, Sec-CH-UA-Platform-Version
Content-Security-Policy: require-trusted-types-for 'script';report-uri /_/DriveUntrustedContentHttp/cspreport, script-src 'report-sample' 'nonce-zmd1QuuOzRweP0P5by4AEw' 'unsafe-inline';object-src 'none';base-uri 'self';report-uri /_/DriveUntrustedContentHttp/cspreport;worker-src 'self'
Permissions-Policy: ch-ua-arch=*, ch-ua-bitness=*, ch-ua-full-version=*, ch-ua-full-version-list=*, ch-ua-model=*, ch-ua-wow64=*, ch-ua-form-factors=*, ch-ua-platform=*, ch-ua-platform-version=*
Cross-Origin-Opener-Policy: same-origin
Content-Length: 1642
X-GUploader-UploadID: AD-8ljuRuB1NNgmIhT5ps_eE4FIket8cSeheb8dr5WVTya-Z7OF3C171ztUmVAJ59WI9dIshf68
Server: UploadServer
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Content-Security-Policy: sandbox allow-scripts

""")
Stacktrace:
  [1] (::HTTP.ConnectionRequest.var"#connections#4"{…})(req::HTTP.Messages.Request; proxy::Nothing, socket_type::Type, socket_type_tls::Nothing, readtimeout::Int64, connect_timeout::Int64, logerrors::Bool, logtag::Nothing, closeimmediately::Bool, kw::@Kwargs{…})
    @ HTTP.ConnectionRequest ~/.julia/packages/HTTP/sJD5V/src/clientlayers/ConnectionRequest.jl:144
  [2] connections
    @ ~/.julia/packages/HTTP/sJD5V/src/clientlayers/ConnectionRequest.jl:60 [inlined]
  [3] (::Base.var"#96#98"{…})(args::HTTP.Messages.Request; kwargs::@Kwargs{…})
    @ Base ./error.jl:296
  [4] (::HTTP.RetryRequest.var"#manageretries#3"{…})(req::HTTP.Messages.Request; retry::Bool, retries::Int64, retry_delays::ExponentialBackOff, retry_check::Function, retry_non_idempotent::Bool, kw::@Kwargs{…})
    @ HTTP.RetryRequest ~/.julia/packages/HTTP/sJD5V/src/clientlayers/RetryRequest.jl:75
  [5] manageretries
    @ ~/.julia/packages/HTTP/sJD5V/src/clientlayers/RetryRequest.jl:30 [inlined]
  [6] (::HTTP.CookieRequest.var"#managecookies#4"{…})(req::HTTP.Messages.Request; cookies::Bool, cookiejar::HTTP.Cookies.CookieJar, kw::@Kwargs{…})
    @ HTTP.CookieRequest ~/.julia/packages/HTTP/sJD5V/src/clientlayers/CookieRequest.jl:42
  [7] (::HTTP.HeadersRequest.var"#defaultheaders#2"{…})(req::HTTP.Messages.Request; iofunction::Function, decompress::Bool, basicauth::Bool, detect_content_type::Bool, canonicalize_headers::Bool, kw::@Kwargs{…})
    @ HTTP.HeadersRequest ~/.julia/packages/HTTP/sJD5V/src/clientlayers/HeadersRequest.jl:71
  [8] defaultheaders
    @ ~/.julia/packages/HTTP/sJD5V/src/clientlayers/HeadersRequest.jl:14 [inlined]
  [9] (::HTTP.RedirectRequest.var"#redirects#3"{…})(req::HTTP.Messages.Request; redirect::Bool, redirect_limit::Int64, redirect_method::Nothing, forwardheaders::Bool, response_stream::Nothing, kw::@Kwargs{…})
    @ HTTP.RedirectRequest ~/.julia/packages/HTTP/sJD5V/src/clientlayers/RedirectRequest.jl:25
 [10] redirects
    @ ~/.julia/packages/HTTP/sJD5V/src/clientlayers/RedirectRequest.jl:14 [inlined]
 [11] (::HTTP.MessageRequest.var"#makerequest#3"{…})(method::String, url::URIs.URI, headers::Vector{…}, body::Nothing; copyheaders::Bool, response_stream::Nothing, http_version::HTTP.Strings.HTTPVersion, verbose::Int64, kw::@Kwargs{…})
    @ HTTP.MessageRequest ~/.julia/packages/HTTP/sJD5V/src/clientlayers/MessageRequest.jl:35
 [12] makerequest
    @ ~/.julia/packages/HTTP/sJD5V/src/clientlayers/MessageRequest.jl:24 [inlined]
 [13] request(stack::HTTP.MessageRequest.var"#makerequest#3"{…}, method::String, url::SubString{…}, h::Vector{…}, b::Nothing, q::Nothing; headers::Vector{…}, body::Nothing, query::Nothing, kw::@Kwargs{…})
    @ HTTP ~/.julia/packages/HTTP/sJD5V/src/HTTP.jl:457
 [14] #request#20
    @ ~/.julia/packages/HTTP/sJD5V/src/HTTP.jl:315 [inlined]
 [15] open
    @ ~/.julia/packages/HTTP/sJD5V/src/HTTP.jl:579 [inlined]
 [16] download(url::SubString{…}, local_path::String, headers::Vector{…}; update_period::Float32, kw::@Kwargs{…})
    @ HTTP ~/.julia/packages/HTTP/sJD5V/src/download.jl:106
 [17] download (repeats 2 times)
    @ ~/.julia/packages/HTTP/sJD5V/src/download.jl:96 [inlined]
 [18] #fetch_http#26
    @ ~/.julia/packages/DataDeps/Y2lje/src/fetch_helpers.jl:80 [inlined]
 [19] fetch_http(remotepath::SubString{String}, localdir::String)
    @ DataDeps ~/.julia/packages/DataDeps/Y2lje/src/fetch_helpers.jl:78
 [20] google_download(url::String, localdir::String)
    @ GoogleDrive ~/.julia/packages/GoogleDrive/z9zIO/src/GoogleDrive.jl:80
 [21] download_ncfile(TMIversion::String)
    @ TMI ~/Downloads/TMI.jl/src/config.jl:81
 [22] config_from_nc(TMIversion::String; compute_lu::Bool)
    @ TMI ~/Downloads/TMI.jl/src/config.jl:15
 [23] config_from_nc(TMIversion::String)
    @ TMI ~/Downloads/TMI.jl/src/config.jl:13
 [24] top-level scope
    @ REPL[13]:1

caused by: HTTP.Exceptions.StatusError(404, "GET", "/download?id=1ADkDI3Fc3z4Vm75K5u6hx0Yu1P0iVnW1&export=download", HTTP.Messages.Response:
"""
HTTP/1.1 404 Not Found
Content-Type: text/html; charset=utf-8
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Pragma: no-cache
Expires: Mon, 01 Jan 1990 00:00:00 GMT
Date: Wed, 04 Sep 2024 16:50:16 GMT
Accept-CH: Sec-CH-UA-Arch, Sec-CH-UA-Bitness, Sec-CH-UA-Full-Version, Sec-CH-UA-Full-Version-List, Sec-CH-UA-Model, Sec-CH-UA-WoW64, Sec-CH-UA-Form-Factors, Sec-CH-UA-Platform, Sec-CH-UA-Platform-Version
Content-Security-Policy: require-trusted-types-for 'script';report-uri /_/DriveUntrustedContentHttp/cspreport, script-src 'report-sample' 'nonce-zmd1QuuOzRweP0P5by4AEw' 'unsafe-inline';object-src 'none';base-uri 'self';report-uri /_/DriveUntrustedContentHttp/cspreport;worker-src 'self'
Permissions-Policy: ch-ua-arch=*, ch-ua-bitness=*, ch-ua-full-version=*, ch-ua-full-version-list=*, ch-ua-model=*, ch-ua-wow64=*, ch-ua-form-factors=*, ch-ua-platform=*, ch-ua-platform-version=*
Cross-Origin-Opener-Policy: same-origin
Content-Length: 1642
X-GUploader-UploadID: AD-8ljuRuB1NNgmIhT5ps_eE4FIket8cSeheb8dr5WVTya-Z7OF3C171ztUmVAJ59WI9dIshf68
Server: UploadServer
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Content-Security-Policy: sandbox allow-scripts

""")
Stacktrace:
  [1] (::HTTP.ExceptionRequest.var"#exceptions#2"{…})(stream::HTTP.Streams.Stream{…}; status_exception::Bool, timedout::Nothing, logerrors::Bool, logtag::Nothing, kw::@Kwargs{…})
    @ HTTP.ExceptionRequest ~/.julia/packages/HTTP/sJD5V/src/clientlayers/ExceptionRequest.jl:19
  [2] exceptions
    @ ~/.julia/packages/HTTP/sJD5V/src/clientlayers/ExceptionRequest.jl:13 [inlined]
  [3] (::HTTP.TimeoutRequest.var"#timeouts#3"{…})(stream::HTTP.Streams.Stream{…}; readtimeout::Int64, logerrors::Bool, logtag::Nothing, kw::@Kwargs{…})
    @ HTTP.TimeoutRequest ~/.julia/packages/HTTP/sJD5V/src/clientlayers/TimeoutRequest.jl:18
  [4] (::HTTP.ConnectionRequest.var"#connections#4"{…})(req::HTTP.Messages.Request; proxy::Nothing, socket_type::Type, socket_type_tls::Nothing, readtimeout::Int64, connect_timeout::Int64, logerrors::Bool, logtag::Nothing, closeimmediately::Bool, kw::@Kwargs{…})
    @ HTTP.ConnectionRequest ~/.julia/packages/HTTP/sJD5V/src/clientlayers/ConnectionRequest.jl:122
  [5] connections
    @ ~/.julia/packages/HTTP/sJD5V/src/clientlayers/ConnectionRequest.jl:60 [inlined]
  [6] (::Base.var"#96#98"{…})(args::HTTP.Messages.Request; kwargs::@Kwargs{…})
    @ Base ./error.jl:296
  [7] (::HTTP.RetryRequest.var"#manageretries#3"{…})(req::HTTP.Messages.Request; retry::Bool, retries::Int64, retry_delays::ExponentialBackOff, retry_check::Function, retry_non_idempotent::Bool, kw::@Kwargs{…})
    @ HTTP.RetryRequest ~/.julia/packages/HTTP/sJD5V/src/clientlayers/RetryRequest.jl:75
  [8] manageretries
    @ ~/.julia/packages/HTTP/sJD5V/src/clientlayers/RetryRequest.jl:30 [inlined]
  [9] (::HTTP.CookieRequest.var"#managecookies#4"{…})(req::HTTP.Messages.Request; cookies::Bool, cookiejar::HTTP.Cookies.CookieJar, kw::@Kwargs{…})
    @ HTTP.CookieRequest ~/.julia/packages/HTTP/sJD5V/src/clientlayers/CookieRequest.jl:42
 [10] (::HTTP.HeadersRequest.var"#defaultheaders#2"{…})(req::HTTP.Messages.Request; iofunction::Function, decompress::Bool, basicauth::Bool, detect_content_type::Bool, canonicalize_headers::Bool, kw::@Kwargs{…})
    @ HTTP.HeadersRequest ~/.julia/packages/HTTP/sJD5V/src/clientlayers/HeadersRequest.jl:71
 [11] defaultheaders
    @ ~/.julia/packages/HTTP/sJD5V/src/clientlayers/HeadersRequest.jl:14 [inlined]
 [12] (::HTTP.RedirectRequest.var"#redirects#3"{…})(req::HTTP.Messages.Request; redirect::Bool, redirect_limit::Int64, redirect_method::Nothing, forwardheaders::Bool, response_stream::Nothing, kw::@Kwargs{…})
    @ HTTP.RedirectRequest ~/.julia/packages/HTTP/sJD5V/src/clientlayers/RedirectRequest.jl:25
 [13] redirects
    @ ~/.julia/packages/HTTP/sJD5V/src/clientlayers/RedirectRequest.jl:14 [inlined]
 [14] (::HTTP.MessageRequest.var"#makerequest#3"{…})(method::String, url::URIs.URI, headers::Vector{…}, body::Nothing; copyheaders::Bool, response_stream::Nothing, http_version::HTTP.Strings.HTTPVersion, verbose::Int64, kw::@Kwargs{…})
    @ HTTP.MessageRequest ~/.julia/packages/HTTP/sJD5V/src/clientlayers/MessageRequest.jl:35
 [15] makerequest
    @ ~/.julia/packages/HTTP/sJD5V/src/clientlayers/MessageRequest.jl:24 [inlined]
 [16] request(stack::HTTP.MessageRequest.var"#makerequest#3"{…}, method::String, url::SubString{…}, h::Vector{…}, b::Nothing, q::Nothing; headers::Vector{…}, body::Nothing, query::Nothing, kw::@Kwargs{…})
    @ HTTP ~/.julia/packages/HTTP/sJD5V/src/HTTP.jl:457
 [17] #request#20
    @ ~/.julia/packages/HTTP/sJD5V/src/HTTP.jl:315 [inlined]
 [18] open
    @ ~/.julia/packages/HTTP/sJD5V/src/HTTP.jl:579 [inlined]
 [19] download(url::SubString{…}, local_path::String, headers::Vector{…}; update_period::Float32, kw::@Kwargs{…})
    @ HTTP ~/.julia/packages/HTTP/sJD5V/src/download.jl:106
 [20] download (repeats 2 times)
    @ ~/.julia/packages/HTTP/sJD5V/src/download.jl:96 [inlined]
 [21] #fetch_http#26
    @ ~/.julia/packages/DataDeps/Y2lje/src/fetch_helpers.jl:80 [inlined]
 [22] fetch_http(remotepath::SubString{String}, localdir::String)
    @ DataDeps ~/.julia/packages/DataDeps/Y2lje/src/fetch_helpers.jl:78
 [23] google_download(url::String, localdir::String)
    @ GoogleDrive ~/.julia/packages/GoogleDrive/z9zIO/src/GoogleDrive.jl:80
 [24] download_ncfile(TMIversion::String)
    @ TMI ~/Downloads/TMI.jl/src/config.jl:81
 [25] config_from_nc(TMIversion::String; compute_lu::Bool)
    @ TMI ~/Downloads/TMI.jl/src/config.jl:15
 [26] config_from_nc(TMIversion::String)
    @ TMI ~/Downloads/TMI.jl/src/config.jl:13
 [27] top-level scope
    @ REPL[13]:1
Some type information was truncated. Use `show(err)` to see complete types.
b-r-hamilton commented 2 months ago

My current workaround for the TMI_modern_180x90x33_GH11_GH12 is to copy the .nc file from a different installed version of TMI to the version being used where I need it. e.g. brynn@brynn-ThinkPad-T14-Gen-2i:~/.julia/packages/TMI/NUh7w$ cp data/TMI_modern_180x90x33_GH11_GH12.nc ../ynBgR/data/

ggebbie commented 2 months ago

Thank you for noting these errors. It is the same problem (not properly downloaded due to virus scanner) as noted above, I believe.

ggebbie commented 2 months ago

also see Issue #41