I found an issue where SatelliteToolbox's read_tle function does not close the files it opens. If you try to read more than 1024 (at least on my computer), Julia runs out of file descriptors and errors out. It appears that this limit is set at the OS level and can be checked on different platforms as described here. On my Ubuntu 21.04 laptop, I saw an open-files limit of 1024 per process, so I bet that's a pretty common setting.
Easy fix though! Just have the close every TLE file that's opened. I've put some tests below that should replicate the issue and show the fix.
Reproducing issue with SatelliteToolbox v0.9.1
Calling read_tle on many file paths causes Julia to run out of file descriptors and throw an error. Here's how I'm able to force that to happen:
(@v1.6) pkg> activate test
Activating environment at `~/test/Project.toml`
(test) pkg> status SatelliteToolbox
Status `~/test/Project.toml`
[6ac157d9] SatelliteToolbox v0.9.1
julia> using SatelliteToolbox
julia> tledir = "/path/to/tles/" # directory with > 1000 .tle files
"/path/to/tles/"
julia> map(read_tle, readdir(tledir, join=true)[1:1500])
ERROR: SystemError: opening file "/path/to/tles/isszarya_20227_729.tle": Too many open files
Stacktrace:
[1] systemerror(p::String, errno::Int32; extrainfo::Nothing)
@ Base ./error.jl:168
[2] #systemerror#62
@ ./error.jl:167 [inlined]
[3] systemerror
@ ./error.jl:167 [inlined]
[4] open(fname::String; lock::Bool, read::Bool, write::Nothing, create::Nothing, truncate::Nothing, append::Nothing)
@ Base ./iostream.jl:293
[5] open(fname::String, mode::String; lock::Bool)
@ Base ./iostream.jl:355
[6] open
@ ./iostream.jl:355 [inlined]
[7] read_tle (repeats 2 times)
@ ~/.julia/packages/SatelliteToolbox/Ri3Is/src/submodules/SatelliteToolboxTLE/main.jl:118 [inlined]
[8] iterate
@ ./generator.jl:47 [inlined]
[9] collect_to!(dest::Vector{Vector{TLE}}, itr::Base.Generator{Vector{String}, typeof(read_tle)}, offs::Int64, st::Int64)
@ Base ./array.jl:724
[10] collect_to_with_first!
@ ./array.jl:702 [inlined]
[11] _collect(c::Vector{String}, itr::Base.Generator{Vector{String}, typeof(read_tle)}, #unused#::Base.EltypeUnknown, isz::Base.HasShape{1})
@ Base ./array.jl:696
[12] collect_similar(cont::Vector{String}, itr::Base.Generator{Vector{String}, typeof(read_tle)})
@ Base ./array.jl:606
[13] map(f::Function, A::Vector{String})
@ Base ./abstractarray.jl:2294
[14] top-level scope
@ REPL[6]:1
julia>
Running lsof -c julia at this point also shows tons of open .tle files.
Same test with this patch
(@v1.6) pkg> activate test
Activating environment at `~/test/Project.toml`
(test) pkg> add https://github.com/ThatcherC/SatelliteToolbox.jl#close-tle
Updating git-repo `https://github.com/ThatcherC/SatelliteToolbox.jl`
Updating registry at `~/.julia/registries/General`
Updating git-repo `https://github.com/JuliaRegistries/General.git`
Resolving package versions...
...
(test) pkg> status SatelliteToolbox
Status `~/test/Project.toml`
[6ac157d9] SatelliteToolbox v0.9.1 `https://github.com/ThatcherC/SatelliteToolbox.jl#close-tle`
julia> using SatelliteToolbox
julia> tledir = "/path/to/tles/"
"/path/to/tles/"
julia> map(read_tle, readdir(tledir, join=true)[1:1500]) # takes a while but works
1500-element Vector{Vector{TLE}}:
[TLE: CAPELLA-1 (Epoch = 2021-01-12T19:17:48.172)]
[TLE: CAPELLA-1 (Epoch = 2021-01-12T20:53:55.629)]
...
julia>
I found an issue where SatelliteToolbox's
read_tle
function does not close the files it opens. If you try to read more than 1024 (at least on my computer), Julia runs out of file descriptors and errors out. It appears that this limit is set at the OS level and can be checked on different platforms as described here. On my Ubuntu 21.04 laptop, I saw an open-files limit of 1024 per process, so I bet that's a pretty common setting.Easy fix though! Just have the close every TLE file that's opened. I've put some tests below that should replicate the issue and show the fix.
Reproducing issue with SatelliteToolbox v0.9.1
Calling
read_tle
on many file paths causes Julia to run out of file descriptors and throw an error. Here's how I'm able to force that to happen:Running
lsof -c julia
at this point also shows tons of open.tle
files.Same test with this patch
Works! No
Too many open files
errors.