Open rupertoverall opened 7 months ago
Marielamend
commented
7 months ago Hi @rupertoverall!
Sorry you ran into this issue! Here is a previous issue that was resolved that features your same error, Cannot select videos that are in the project #1660 where they increased the limit of open files allowable by the OS.
Let us know if this works! Best, Mariela
hlanovoi
commented
6 months ago Hi all, I had the same issue and my previously saved sessions (I had two open) were deleted. I still have the predictions and models folders for both but the .slp file containing my many labeled frames have been deleted. Are these files lost permanently and is there any way to build on the models from these projects? I tried to train one of these prior models in a new project and it looks like it's reverted back to the few labels in this new project, this will be quite a loss of effort for me, any recommendations?
talmo
commented
6 months ago Hi @hlanovoi,
In the model folders there are some SLP files which store a cached copy of the labels used to train those models. You can try to open those in the GUI and/or import and merge them into the new project.
Let us know if you need some help with that.
In general, we also strongly recommend saving out new versions of your labels as often as possible by going to File --> Save as... and saving a new version (which SLEAP automatically increments in the filename).
hlanovoi
commented
6 months ago Hello @talmo
Thank you for your reply, so from what it looks like I by going to file-> merge into project I can maybe select one of the files you are talking derived from the model with the most labels.
Within the folder for this model I see: lables_pr.val.slp labels_pr.train.slp labels_gt.val.slp labels_gt.train.slp
Is this the right approach and which one of these is most appropriate to import?
I'm noting this point about file hygiene, thanks so much!
talmo
commented
6 months ago You'd want to import the labels_gt.val.slp and the labels_gt.train.slp, since those contain the ground truth labels used to train.
Cheers,
Talmo
Bug description
Adding new files often throws an error 'Too many open files' which crashes the session.
Expected behaviour
Adding new videos should add new videos; and leave the session in a sane state (i.e. able to be saved). If there is insufficient memory or other resource, the action should fail with an error and remain in the previous state.
Actual behaviour
An error is thrown which leaves the session unuseable. Closing the session is not clean and actually deletes(!) the previously saved session file.
Your personal set up
SLEAP: 1.3.3 TensorFlow: 2.9.2 Numpy: 1.22.4 Python: 3.9.15 OS: macOS-13.6.6-arm64-arm-64bit
Run on an Apple (M1) laptop with limited RAM, so the issue could be triggered by resource limitations.
Environment packages
``` # Name Version Build Channel abseil-cpp 20211102.0 he4e09e4_3 conda-forge absl-py 1.4.0 pypi_0 pypi aiohttp 3.9.3 py39h17cfd9d_0 conda-forge aiosignal 1.3.1 pyhd8ed1ab_0 conda-forge aom 3.5.0 h7ea286d_0 conda-forge astunparse 1.6.3 pyhd8ed1ab_0 conda-forge async-timeout 4.0.3 pyhd8ed1ab_0 conda-forge attrs 23.2.0 pyh71513ae_0 conda-forge blinker 1.7.0 pyhd8ed1ab_0 conda-forge blosc 1.21.5 hc338f07_0 conda-forge brotli 1.0.9 h1a8c8d9_9 conda-forge brotli-bin 1.0.9 h1a8c8d9_9 conda-forge brotli-python 1.0.9 py39h23fbdae_9 conda-forge brunsli 0.1 h9f76cd9_0 conda-forge bzip2 1.0.8 h93a5062_5 conda-forge c-ares 1.26.0 h93a5062_0 conda-forge c-blosc2 2.13.2 ha57e6be_0 conda-forge ca-certificates 2024.2.2 hf0a4a13_0 conda-forge cached-property 1.5.2 hd8ed1ab_1 conda-forge cached_property 1.5.2 pyha770c72_1 conda-forge cachetools 5.3.1 pypi_0 pypi cairo 1.16.0 h73a0509_1014 conda-forge cattrs 1.1.1 pyhd8ed1ab_0 conda-forge certifi 2024.2.2 pyhd8ed1ab_0 conda-forge cffi 1.16.0 py39he153c15_0 conda-forge cfitsio 4.2.0 h2f961c4_0 conda-forge charls 2.3.4 hbdafb3b_0 conda-forge charset-normalizer 3.2.0 pypi_0 pypi click 8.1.7 unix_pyh707e725_0 conda-forge contourpy 1.2.0 py39he9de807_0 conda-forge cryptography 39.0.0 py39haa0b8cc_0 conda-forge cycler 0.12.1 pyhd8ed1ab_0 conda-forge dav1d 1.2.1 hb547adb_0 conda-forge efficientnet 1.0.0 pypi_0 pypi expat 2.5.0 hb7217d7_1 conda-forge ffmpeg 4.4.2 gpl_hf318d42_112 conda-forge font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge font-ttf-inconsolata 3.000 h77eed37_0 conda-forge font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge font-ttf-ubuntu 0.83 h77eed37_1 conda-forge fontconfig 2.14.2 h82840c6_0 conda-forge fonts-conda-ecosystem 1 0 conda-forge fonts-conda-forge 1 0 conda-forge fonttools 4.49.0 py39h17cfd9d_0 conda-forge freetype 2.12.1 hadb7bae_2 conda-forge frozenlist 1.4.1 py39h17cfd9d_0 conda-forge gast 0.4.0 pyh9f0ad1d_0 conda-forge geos 3.12.1 h965bd2d_0 conda-forge gettext 0.21.1 h0186832_0 conda-forge giflib 5.2.1 h1a8c8d9_3 conda-forge glib 2.78.4 h1059232_0 conda-forge glib-tools 2.78.4 h1059232_0 conda-forge gmp 6.3.0 h965bd2d_0 conda-forge gnutls 3.7.9 hd26332c_0 conda-forge google-auth 2.23.0 pypi_0 pypi google-auth-oauthlib 0.4.6 pyhd8ed1ab_0 conda-forge google-pasta 0.2.0 pyh8c360ce_0 conda-forge graphite2 1.3.13 h9f76cd9_1001 conda-forge grpc-cpp 1.46.4 hcaf9be7_3 conda-forge grpcio 1.58.0 pypi_0 pypi gst-plugins-base 1.22.9 h09b4b5e_0 conda-forge gstreamer 1.22.9 h551c6ff_0 conda-forge h5py 3.8.0 nompi_py39hc9149d8_100 conda-forge harfbuzz 5.3.0 hddbc195_0 conda-forge hdf5 1.12.2 nompi_h55deafc_101 conda-forge hdmf 3.9.0 pypi_0 pypi icu 70.1 h6b3803e_0 conda-forge idna 3.4 pypi_0 pypi image-classifiers 1.0.0 pypi_0 pypi imagecodecs 2022.9.26 py39hd7f743f_4 conda-forge imageio 2.34.0 pyh4b66e23_0 conda-forge imgaug 0.4.0 pyhd8ed1ab_1 conda-forge imgstore 0.2.9 pypi_0 pypi importlib-metadata 7.0.1 pyha770c72_0 conda-forge importlib-resources 6.1.1 pyhd8ed1ab_0 conda-forge importlib_resources 6.1.1 pyhd8ed1ab_0 conda-forge jasper 2.0.33 hc3cd1e9_1 conda-forge joblib 1.3.2 pyhd8ed1ab_0 conda-forge jpeg 9e h1a8c8d9_3 conda-forge jsmin 3.0.1 pyhd8ed1ab_0 conda-forge jsonpickle 1.2 py_0 conda-forge jsonschema 4.19.0 pypi_0 pypi jsonschema-specifications 2023.7.1 pypi_0 pypi jxrlib 1.1 h93a5062_3 conda-forge keras 2.9.0 pyhd8ed1ab_0 conda-forge keras-applications 1.0.8 pypi_0 pypi keras-preprocessing 1.1.2 pyhd8ed1ab_0 conda-forge kiwisolver 1.4.5 py39hbd775c9_1 conda-forge krb5 1.20.1 h127bd45_0 conda-forge lame 3.100 h1a8c8d9_1003 conda-forge lazy_loader 0.3 pyhd8ed1ab_0 conda-forge lcms2 2.14 h8193b64_0 conda-forge lerc 4.0.0 h9a09cb3_0 conda-forge libabseil 20211102.0 cxx17_h28b99d4_3 conda-forge libaec 1.1.2 h13dd4ca_1 conda-forge libavif 0.11.1 h9f83d30_2 conda-forge libblas 3.9.0 20_osxarm64_openblas conda-forge libbrotlicommon 1.0.9 h1a8c8d9_9 conda-forge libbrotlidec 1.0.9 h1a8c8d9_9 conda-forge libbrotlienc 1.0.9 h1a8c8d9_9 conda-forge libcblas 3.9.0 20_osxarm64_openblas conda-forge libclang 16.0.6 pypi_0 pypi libclang13 14.0.6 default_hc7183e1_1 conda-forge libcurl 7.87.0 hbe9bab4_0 conda-forge libcxx 16.0.6 h4653b0c_0 conda-forge libdeflate 1.14 h1a8c8d9_0 conda-forge libedit 3.1.20191231 hc8eb9b7_2 conda-forge libev 4.33 h93a5062_2 conda-forge libexpat 2.5.0 hb7217d7_1 conda-forge libffi 3.4.2 h3422bc3_5 conda-forge libgfortran 5.0.0 13_2_0_hd922786_3 conda-forge libgfortran5 13.2.0 hf226fd6_3 conda-forge libglib 2.78.4 h1635a5e_0 conda-forge libiconv 1.17 h0d3ecfb_2 conda-forge libidn2 2.3.7 h93a5062_0 conda-forge liblapack 3.9.0 20_osxarm64_openblas conda-forge liblapacke 3.9.0 20_osxarm64_openblas conda-forge libllvm14 14.0.6 hd1a9a77_4 conda-forge libnghttp2 1.51.0 hd184df1_0 conda-forge libogg 1.3.4 h27ca646_1 conda-forge libopenblas 0.3.25 openmp_h6c19121_0 conda-forge libopencv 4.6.0 py39he1c1adf_3 conda-forge libopus 1.3.1 h27ca646_1 conda-forge libpng 1.6.42 h091b4b1_0 conda-forge libpq 15.1 hbce9e56_3 conda-forge libprotobuf 3.20.3 hb5ab8b9_0 conda-forge libsodium 1.0.18 h27ca646_1 conda-forge libsqlite 3.45.1 h091b4b1_0 conda-forge libssh2 1.10.0 hb80f160_3 conda-forge libtasn1 4.19.0 h1a8c8d9_0 conda-forge libtiff 4.4.0 heb92581_5 conda-forge libunistring 0.9.10 h3422bc3_0 conda-forge libvorbis 1.3.7 h9f76cd9_0 conda-forge libvpx 1.11.0 hc470f4d_3 conda-forge libwebp-base 1.3.2 hb547adb_0 conda-forge libxcb 1.13 h9b22ae9_1004 conda-forge libxml2 2.10.3 h67585b2_4 conda-forge libxslt 1.1.37 h1bd8bc4_0 conda-forge libzlib 1.2.13 h53f4e23_5 conda-forge libzopfli 1.0.3 h9f76cd9_0 conda-forge llvm-openmp 17.0.6 hcd81f8e_0 conda-forge lz4-c 1.9.4 hb7217d7_0 conda-forge markdown 3.4.4 pypi_0 pypi markdown-it-py 3.0.0 pyhd8ed1ab_0 conda-forge markupsafe 2.1.3 pypi_0 pypi matplotlib-base 3.8.3 py39hbab7938_0 conda-forge mdurl 0.1.2 pyhd8ed1ab_0 conda-forge multidict 6.0.5 py39h02fc5c5_0 conda-forge munkres 1.1.4 pyh9f0ad1d_0 conda-forge mysql-common 8.0.32 hab468bb_0 conda-forge mysql-libs 8.0.32 hea58576_0 conda-forge ncurses 6.4 h463b476_2 conda-forge ndx-pose 0.1.1 pypi_0 pypi nettle 3.9.1 h40ed0f5_0 conda-forge networkx 3.2.1 pyhd8ed1ab_0 conda-forge nixio 1.5.3 pypi_0 pypi nspr 4.35 hb7217d7_0 conda-forge nss 3.98 h5ce2875_0 conda-forge numpy 1.22.4 py39h7df2422_0 conda-forge oauthlib 3.2.2 pyhd8ed1ab_0 conda-forge opencv 4.6.0 py39hdf13c20_3 conda-forge openh264 2.3.1 hb7217d7_2 conda-forge openjpeg 2.5.0 h5d4e404_1 conda-forge openssl 1.1.1w h53f4e23_0 conda-forge opt_einsum 3.3.0 pyhc1e730c_2 conda-forge p11-kit 0.24.1 h29577a5_0 conda-forge packaging 23.2 pyhd8ed1ab_0 conda-forge pandas 2.2.0 py39h47e51b9_0 conda-forge patsy 0.5.6 pyhd8ed1ab_0 conda-forge pcre2 10.42 h26f9a81_0 conda-forge pillow 9.2.0 py39h139752e_3 conda-forge pip 24.0 pyhd8ed1ab_0 conda-forge pixman 0.43.2 hebf3989_0 conda-forge protobuf 3.19.6 pypi_0 pypi psutil 5.9.8 py39h17cfd9d_0 conda-forge pthread-stubs 0.4 h27ca646_1001 conda-forge py-opencv 4.6.0 py39hfa6204d_3 conda-forge pyasn1 0.5.0 pypi_0 pypi pyasn1-modules 0.3.0 pyhd8ed1ab_0 conda-forge pycparser 2.21 pyhd8ed1ab_0 conda-forge pygments 2.17.2 pyhd8ed1ab_0 conda-forge pyjwt 2.8.0 pyhd8ed1ab_1 conda-forge pykalman 0.9.5 py_1 conda-forge pynwb 2.5.0 pypi_0 pypi pyopenssl 23.2.0 pyhd8ed1ab_1 conda-forge pyparsing 3.1.1 pyhd8ed1ab_0 conda-forge pyside2 5.15.8 py39h0adaba8_2 conda-forge pysocks 1.7.1 pyha2e5f31_6 conda-forge python 3.9.15 h2d96c93_0_cpython conda-forge python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge python-flatbuffers 1.12 pyhd8ed1ab_1 conda-forge python-rapidjson 1.14 py39hf3050f2_0 conda-forge python-tzdata 2024.1 pyhd8ed1ab_0 conda-forge python_abi 3.9 4_cp39 conda-forge pytz 2024.1 pyhd8ed1ab_0 conda-forge pyu2f 0.1.5 pyhd8ed1ab_0 conda-forge pywavelets 1.4.1 py39hf4a74a7_1 conda-forge pyyaml 6.0.1 py39h0f82c59_1 conda-forge pyzmq 25.1.2 py39he1e2164_0 conda-forge qimage2ndarray 1.10.0 pypi_0 pypi qt-main 5.15.8 hfe8d25c_6 conda-forge qtpy 2.4.1 pyhd8ed1ab_0 conda-forge re2 2022.06.01 h9a09cb3_1 conda-forge readline 8.2 h92ec313_1 conda-forge referencing 0.30.2 pypi_0 pypi requests 2.31.0 pyhd8ed1ab_0 conda-forge requests-oauthlib 1.3.1 pyhd8ed1ab_0 conda-forge rich 13.7.0 pyhd8ed1ab_0 conda-forge rpds-py 0.10.3 pypi_0 pypi rsa 4.9 pyhd8ed1ab_0 conda-forge ruamel-yaml 0.17.32 pypi_0 pypi ruamel-yaml-clib 0.2.7 pypi_0 pypi scikit-image 0.22.0 py39hf8cecc8_2 conda-forge scikit-learn 1.0 py39h12ba089_1 conda-forge scikit-video 1.1.11 pyh24bf2e0_0 conda-forge scipy 1.9.0 py39h14896cb_0 conda-forge seaborn 0.13.2 hd8ed1ab_0 conda-forge seaborn-base 0.13.2 pyhd8ed1ab_0 conda-forge segmentation-models 1.0.1 pypi_0 pypi setuptools 69.1.0 pyhd8ed1ab_0 conda-forge shapely 2.0.2 py39ha70ab96_1 conda-forge six 1.15.0 pypi_0 pypi sleap 1.3.3 pypi_0 pypi snappy 1.1.10 h17c5cce_0 conda-forge sqlite 3.45.1 hf2abe2d_0 conda-forge statsmodels 0.14.1 py39h373d45f_0 conda-forge svt-av1 1.4.1 h7ea286d_0 conda-forge tensorboard 2.9.1 pypi_0 pypi tensorboard-data-server 0.6.1 py39haa0b8cc_4 conda-forge tensorboard-plugin-wit 1.8.1 pyhd8ed1ab_0 conda-forge tensorflow 2.9.1 cpu_py39h2839aeb_0 conda-forge tensorflow-base 2.9.1 cpu_py39ha1ad4ae_0 conda-forge tensorflow-estimator 2.9.1 cpu_py39h7b621ec_0 conda-forge tensorflow-hub 0.12.0 pyhca92ed8_0 conda-forge tensorflow-macos 2.9.2 pypi_0 pypi tensorflow-metal 0.5.0 pypi_0 pypi termcolor 2.3.0 pypi_0 pypi threadpoolctl 3.3.0 pyhc1e730c_0 conda-forge tifffile 2022.10.10 pyhd8ed1ab_0 conda-forge tk 8.6.13 h5083fa2_1 conda-forge typing-extensions 4.9.0 hd8ed1ab_0 conda-forge typing_extensions 4.9.0 pyha770c72_0 conda-forge tzdata 2024a h0c530f3_0 conda-forge tzlocal 5.0.1 pypi_0 pypi unicodedata2 15.1.0 py39h0f82c59_0 conda-forge urllib3 1.26.16 pypi_0 pypi werkzeug 2.3.7 pypi_0 pypi wheel 0.42.0 pyhd8ed1ab_0 conda-forge wrapt 1.15.0 pypi_0 pypi x264 1!164.3095 h57fd34a_2 conda-forge x265 3.5 hbc6ce65_3 conda-forge xorg-libxau 1.0.11 hb547adb_0 conda-forge xorg-libxdmcp 1.1.3 h27ca646_0 conda-forge xz 5.2.6 h57fd34a_0 conda-forge yaml 0.2.5 h3422bc3_2 conda-forge yarl 1.9.4 py39h17cfd9d_0 conda-forge zeromq 4.3.5 h965bd2d_0 conda-forge zfp 1.0.1 ha8f4885_0 conda-forge zipp 3.17.0 pyhd8ed1ab_0 conda-forge zlib 1.2.13 h53f4e23_5 conda-forge zlib-ng 2.0.7 h1a8c8d9_0 conda-forge zstd 1.5.5 h4f39d0f_0 conda-forge ```Logs
``` # paste relevant logs here, if any ```Screenshots
How to reproduce
I can create projects with e.g. 100 files and work with these - the error does not always occur with every session. But removing all of these videos and trying to add 30 different ones causes the crash, so it is also not just the number of videos in the project. I have not yet identified a hard-and-fast limit to the number of videos that are 'too many' (or, indeed, whether it is even the video files that are being referrred to in the error).