search

JuliaLinearAlgebra / MatrixDepot.jl

An Extensible Test Matrix Collection for Julia
http://matrixdepotjl.readthedocs.org/
Other
75 stars 22 forks source link

"no metadata .mtx" error on second access #34

Closed tobydriscoll closed 5 years ago

tobydriscoll commented 5 years ago

When I access a SuiteSparse matrix for the first time, everything is fine. When I quit and restart Julia, though, I get a fatal error. Example:

driscoll@~\> julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.0.3 (2018-12-18)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> using MatrixDepot
include group.jl for user defined matrix generators
verify download of index files...
used remote site is https://sparse.tamu.edu/?per_page=All
populating internal database...

julia> A = matrixdepot("Gset/G24");
downloading: https://sparse.tamu.edu/MM/Gset/G24.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 60541  100 60541    0     0   119k      0 --:--:-- --:--:-- --:--:--  120k
sx G24/G24.mtx
ize(A
julia> size(A)
(2000, 2000)

julia> eltype(A)
Bool

julia>
driscoll@~\> julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.0.3 (2018-12-18)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> using MatrixDepot
include group.jl for user defined matrix generators
verify download of index files...
used remote site is https://sparse.tamu.edu/?per_page=All
populating internal database...

julia> A = matrixdepot("Gset/G24");
ERROR: MatrixDepot.DataError("Gset/G24 has no metadata G24.mtx")
Stacktrace:
 [1] daterr(::String) at /Users/driscoll/.julia/packages/MatrixDepot/E06jU/src/common.jl:6
 [2] _metareader(::MatrixDepot.RemoteMatrixData{MatrixDepot.TURemoteType}, ::String) at /Users/driscoll/.julia/packages/MatrixDepot/E06jU/src/datareader.jl:35
 [3] (::getfield(MatrixDepot, Symbol("##112#113")){MatrixDepot.MatrixDescriptor{MatrixDepot.RemoteMatrixData{MatrixDepot.TURemoteType}}})() at /Users/driscoll/.julia/packages/MatrixDepot/E06jU/src/datareader.jl:12
 [4] get!(::getfield(MatrixDepot, Symbol("##112#113")){MatrixDepot.MatrixDescriptor{MatrixDepot.RemoteMatrixData{MatrixDepot.TURemoteType}}}, ::Dict{Any,Any}, ::String) at ./dict.jl:453
 [5] metareader(::MatrixDepot.MatrixDescriptor{MatrixDepot.RemoteMatrixData{MatrixDepot.TURemoteType}}, ::String) at /Users/driscoll/.julia/packages/MatrixDepot/E06jU/src/datareader.jl:11
 [6] getproperty(::MatrixDepot.MatrixDescriptor{MatrixDepot.RemoteMatrixData{MatrixDepot.TURemoteType}}, ::Symbol) at /Users/driscoll/.julia/packages/MatrixDepot/E06jU/src/datareader.jl:62
 [7] (::getfield(MatrixDepot, Symbol("##86#87")))(::MatrixDepot.MatrixDescriptor{MatrixDepot.RemoteMatrixData{MatrixDepot.TURemoteType}}) at /Users/driscoll/.julia/packages/MatrixDepot/E06jU/src/common.jl:511
 [8] mdopen(::getfield(MatrixDepot, Symbol("##86#87")), ::MatrixDepot.MatrixDatabase, ::String) at /Users/driscoll/.julia/packages/MatrixDepot/E06jU/src/common.jl:457
 [9] matrixdepot at /Users/driscoll/.julia/packages/MatrixDepot/E06jU/src/common.jl:510 [inlined]
 [10] matrixdepot(::String) at /Users/driscoll/.julia/packages/MatrixDepot/E06jU/src/common.jl:508
 [11] top-level scope at none:0

julia> md = mdopen("Gset/G24");

julia> propertynames(md)
4-element Array{Symbol,1}:
 :m
 :n
 :nnz
 :dnz

julia> md.A
ERROR: MatrixDepot.DataError("Gset/G24 has no metadata G24.mtx")
Stacktrace:
 [1] daterr(::String) at /Users/driscoll/.julia/packages/MatrixDepot/E06jU/src/common.jl:6
 [2] _metareader(::MatrixDepot.RemoteMatrixData{MatrixDepot.TURemoteType}, ::String) at /Users/driscoll/.julia/packages/MatrixDepot/E06jU/src/datareader.jl:35
 [3] (::getfield(MatrixDepot, Symbol("##112#113")){MatrixDepot.MatrixDescriptor{MatrixDepot.RemoteMatrixData{MatrixDepot.TURemoteType}}})() at /Users/driscoll/.julia/packages/MatrixDepot/E06jU/src/datareader.jl:12
 [4] get!(::getfield(MatrixDepot, Symbol("##112#113")){MatrixDepot.MatrixDescriptor{MatrixDepot.RemoteMatrixData{MatrixDepot.TURemoteType}}}, ::Dict{Any,Any}, ::String) at ./dict.jl:453
 [5] metareader(::MatrixDepot.MatrixDescriptor{MatrixDepot.RemoteMatrixData{MatrixDepot.TURemoteType}}, ::String) at /Users/driscoll/.julia/packages/MatrixDepot/E06jU/src/datareader.jl:11
 [6] getproperty(::MatrixDepot.MatrixDescriptor{MatrixDepot.RemoteMatrixData{MatrixDepot.TURemoteType}}, ::Symbol) at /Users/driscoll/.julia/packages/MatrixDepot/E06jU/src/datareader.jl:62

julia>
GregVernon commented 5 years ago

I'm having the same issue. I did a bulk load, MatrixDepot.load("*/*/*"), and now can't load any matrices. This is quite concerning.

KlausC commented 5 years ago

@tobydriscoll, @GregVernon , sorry for the late response. I was busy with other projects and did not receive notification.

That is not the expected behavior. The problem is, that the program cannot find file ".../MatrixDepot/data/uf/Gset/G24.mtx" in the installation directory, see: MatrixDepot.DATA_DIR.

The initial download should create the required directories and extract all the files in there. Maybe there is a permission problem. Could you, please, report, what MatrixDepot.DATA_DIR is and the status of the directory.

GregVernon commented 5 years ago

@KlausC , no worries:

julia> MatrixDepot.DATA_DIR
"C:\\Users\\gregj\\.julia\\packages\\MatrixDepot\\E06jU\\data"

julia> uperm(MatrixDepot.DATA_DIR)
0x06

image

KlausC commented 5 years ago

Unfortunately I have no windows installation at hand. Unix user permissions for data I would expect 0x07, while the printout looks fine. On Linux the execute bit for directories is required in order to be allowed to access subdirectories.

Can you tell me what is in data? There should be subdirectories uf and uf\Gset\G24 with proper permissions.

KlausC commented 5 years ago

Maybe it is a good idea to make data/uf and data/mm soft links referring to directories in your home area, where you have full control. That is what I did, in order to keep the precious downloaded data at a save place (some of the files are really big - like 1 GiB).

tobydriscoll commented 5 years ago

macos here.

driscoll@src\> ls -l ~/.julia/packages/MatrixDepot/E06jU/
total 72
-r--r--r--   1 driscoll  WIN\Domain Users   1167 Dec  7 11:30 LICENSE.md
-r--r--r--   1 driscoll  WIN\Domain Users   5459 Dec  7 11:30 NEWS.md
-r--r--r--   1 driscoll  WIN\Domain Users  16920 Dec  7 11:30 README.md
-r--r--r--   1 driscoll  WIN\Domain Users     15 Dec  7 11:30 REQUIRE
drwxr-xr-x   8 driscoll  WIN\Domain Users    256 Jan 15 14:12 data
drwxr-xr-x  20 driscoll  WIN\Domain Users    640 Dec  7 11:30 doc
drwxr-xr-x   4 driscoll  WIN\Domain Users    128 Jan 15 14:11 myMatrixDepot
drwxr-xr-x  14 driscoll  WIN\Domain Users    448 Dec  7 11:30 src
drwxr-xr-x  56 driscoll  WIN\Domain Users   1792 Dec  7 11:30 test
driscoll@src\> ls -l ~/.julia/packages/MatrixDepot/E06jU/data
total 3720
-r--r--r--  1 driscoll  WIN\Domain Users      427 Dec  7 11:30 README.md
-rw-r--r--  1 driscoll  WIN\Domain Users   471128 Jan 15 14:11 db.data
-r--r--r--  1 driscoll  WIN\Domain Users    41539 Dec  7 11:30 mm_matrices.html
-r-xr-xr-x  1 driscoll  WIN\Domain Users      161 Dec  7 11:30 tamufilter
drwxr-xr-x  8 driscoll  WIN\Domain Users      256 Jan 16 14:40 uf
-r--r--r--  1 driscoll  WIN\Domain Users  1373626 Dec  7 11:30 uf_matrices.html
driscoll@src\> ls -l ~/.julia/packages/MatrixDepot/E06jU/data/uf
total 0
drwxr-xr-x  4 driscoll  WIN\Domain Users  128 Jan 15 14:29 Arenas
drwxr-xr-x  4 driscoll  WIN\Domain Users  128 Jan 16 14:46 Gset
drwxr-xr-x  3 driscoll  WIN\Domain Users   96 Jan 16 14:40 Newman
drwxr-xr-x  3 driscoll  WIN\Domain Users   96 Jan 15 14:12 Pajek
drwxr-xr-x  3 driscoll  WIN\Domain Users   96 Jan 15 14:28 SNAP
drwxr-xr-x  3 driscoll  WIN\Domain Users   96 Jan 16 14:39 VDOL
driscoll@src\> ls -l ~/.julia/packages/MatrixDepot/E06jU/data/uf/Gset/
total 0
drwxr-xr-x  3 driscoll  WIN\Domain Users  96 Jan 16 14:37 G12
drwxr-xr-x  3 driscoll  WIN\Domain Users  96 Jan 16 14:46 G24
driscoll@src\> ls -l ~/.julia/packages/MatrixDepot/E06jU/data/uf/Gset/G12
total 40
-rw-------  1 driscoll  WIN\Domain Users  16913 Jan 29  2007 G12.mtx
driscoll@src\> ls -l ~/.julia/packages/MatrixDepot/E06jU/data/uf/Gset/G24
total 352
-rw-------  1 driscoll  WIN\Domain Users  178271 Jan 29  2007 G24.mtx
KlausC commented 5 years ago

@tobydriscoll, the dir and file permissions look fine. Could you compare the output and report:

julia> md = mdopen("*/G24")
(PS Gset/G24(#485)  2000x2000(39980/19990) 1996 [A] 'undirected random graph' [Random matrix, 1% uniformly distributed. G24=pattern of G29]()
julia> MatrixDepot.metadata(md)
1-element Array{AbstractString,1}:
 "G24.mtx"
julia> MatrixDepot.metasymbols(md)
1-element Array{Symbol,1}
 :A

Another attempt: Try again after removing data/db.data, then restart julia.

KlausC commented 5 years ago

I could reproduce the error. It is a bug and will be fixed soon. As a work-around you have to:

md = mdopen("*/G24")
MatrixDepot.addmetadata!(md.data)
tobydriscoll commented 5 years ago

Cool, thanks.

andreasnoack commented 5 years ago

@tobydriscoll Would you be able to try out master to see if the issue has been completely resolved and let us know how it goes?

tobydriscoll commented 5 years ago

I had to first ] rm MatrixDepot, then ] add MatrixDepot#master, and then it works.

willow-ahrens commented 4 years ago

This fixed the bug for me as well. Perhaps this bugfix could get put in a release?