JuliaSpace / SatelliteToolbox.jl

A toolbox for satellite analysis written in julia language.
MIT License
248 stars 33 forks source link

Now closes TLE files that are opened for reading #60

Closed ThatcherC closed 2 years ago

ThatcherC commented 2 years ago

I found an issue where SatelliteToolbox's read_tle function does not close the files it opens. If you try to read more than 1024 (at least on my computer), Julia runs out of file descriptors and errors out. It appears that this limit is set at the OS level and can be checked on different platforms as described here. On my Ubuntu 21.04 laptop, I saw an open-files limit of 1024 per process, so I bet that's a pretty common setting.

Easy fix though! Just have the close every TLE file that's opened. I've put some tests below that should replicate the issue and show the fix.

Reproducing issue with SatelliteToolbox v0.9.1

Calling read_tle on many file paths causes Julia to run out of file descriptors and throw an error. Here's how I'm able to force that to happen:

(@v1.6) pkg> activate test
  Activating environment at `~/test/Project.toml`

(test) pkg> status SatelliteToolbox
      Status `~/test/Project.toml`
  [6ac157d9] SatelliteToolbox v0.9.1

julia> using SatelliteToolbox

julia> tledir = "/path/to/tles/" # directory with > 1000 .tle files
"/path/to/tles/"

julia> map(read_tle, readdir(tledir, join=true)[1:1500])
ERROR: SystemError: opening file "/path/to/tles/isszarya_20227_729.tle": Too many open files
Stacktrace:
  [1] systemerror(p::String, errno::Int32; extrainfo::Nothing)
    @ Base ./error.jl:168
  [2] #systemerror#62
    @ ./error.jl:167 [inlined]
  [3] systemerror
    @ ./error.jl:167 [inlined]
  [4] open(fname::String; lock::Bool, read::Bool, write::Nothing, create::Nothing, truncate::Nothing, append::Nothing)
    @ Base ./iostream.jl:293
  [5] open(fname::String, mode::String; lock::Bool)
    @ Base ./iostream.jl:355
  [6] open
    @ ./iostream.jl:355 [inlined]
  [7] read_tle (repeats 2 times)
    @ ~/.julia/packages/SatelliteToolbox/Ri3Is/src/submodules/SatelliteToolboxTLE/main.jl:118 [inlined]
  [8] iterate
    @ ./generator.jl:47 [inlined]
  [9] collect_to!(dest::Vector{Vector{TLE}}, itr::Base.Generator{Vector{String}, typeof(read_tle)}, offs::Int64, st::Int64)
    @ Base ./array.jl:724
 [10] collect_to_with_first!
    @ ./array.jl:702 [inlined]
 [11] _collect(c::Vector{String}, itr::Base.Generator{Vector{String}, typeof(read_tle)}, #unused#::Base.EltypeUnknown, isz::Base.HasShape{1})
    @ Base ./array.jl:696
 [12] collect_similar(cont::Vector{String}, itr::Base.Generator{Vector{String}, typeof(read_tle)})
    @ Base ./array.jl:606
 [13] map(f::Function, A::Vector{String})
    @ Base ./abstractarray.jl:2294
 [14] top-level scope
    @ REPL[6]:1

julia> 

Running lsof -c julia at this point also shows tons of open .tle files.

Same test with this patch

(@v1.6) pkg> activate test
  Activating environment at `~/test/Project.toml`

(test) pkg> add https://github.com/ThatcherC/SatelliteToolbox.jl#close-tle
    Updating git-repo `https://github.com/ThatcherC/SatelliteToolbox.jl`
    Updating registry at `~/.julia/registries/General`
    Updating git-repo `https://github.com/JuliaRegistries/General.git`
   Resolving package versions...
    ...

(test) pkg> status SatelliteToolbox
      Status `~/test/Project.toml`
  [6ac157d9] SatelliteToolbox v0.9.1 `https://github.com/ThatcherC/SatelliteToolbox.jl#close-tle`

julia> using SatelliteToolbox

julia> tledir = "/path/to/tles/"
"/path/to/tles/"

julia> map(read_tle, readdir(tledir, join=true)[1:1500]) # takes a while but works
1500-element Vector{Vector{TLE}}:
 [TLE: CAPELLA-1 (Epoch = 2021-01-12T19:17:48.172)]
 [TLE: CAPELLA-1 (Epoch = 2021-01-12T20:53:55.629)]
...

julia>

Works! No Too many open files errors.

ronisbr commented 2 years ago

Perfect! Thank you very much!