search

conda-forge / miniforge

A conda-forge distribution.
https://conda-forge.org/miniforge
Other
5.76k stars 306 forks source link

OOM killer on Pi 4 with 1GB memory with "conda update --yes conda" #546

Open wasade opened 5 months ago

wasade commented 5 months ago

Solution to issue cannot be found in the documentation.

Issue

A fresh install of Miniforge was performed on a first boot of Raspberry Pi 4 64-bit with 1GB of memory. Immediately following the call, conda update --yes conda was executed. This resulted in the OOM Killer terminating the process.

Approximately a month ago, the same OS, configuration and hardware were used, with the same steps. In those instances, the OOM Killer was not triggered. I unfortunately cannot assert the specific download of Miniforge on the existing devices other than those downloads and steps occurred in and around December 15th.

While 1GB of memory is not much, it was unexpected as the update previously worked on other devices. I am not sure if 1GB is within the scope of support for Miniforge.

Please let me know if this report would be better for conda/conda. I'm starting here as the build of conda is originating from Miniforge.

Steps taken:

$ curl -L -o miniforge3.sh https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-aarch64.sh
$ chmod +x miniforge3.sh
$ ./miniforge3.sh -b
$ conda update --yes conda

Other information:

$ md5sum miniforge3.sh
cf3e9e4150172729da371bdb49b2fa77  miniforge3.sh
$ uname -a
Linux pmcd-gr 6.1.0-rpi7-rpi-v8 #1 SMP PREEMPT Debian 1:6.1.63-1+rpt1 (2023-11-24) aarch64 GNU/Linux
$ free -h
               total        used        free      shared  buff/cache   available
Mem:           908Mi       131Mi       652Mi       4.0Ki       187Mi       776Mi
Swap:           99Mi        30Mi        69Mi
$ dmesg -T | tail -n 73
[Sat Jan 20 18:56:58 2024] conda invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0
[Sat Jan 20 18:56:58 2024] CPU: 1 PID: 1991 Comm: conda Tainted: G         C         6.1.0-rpi7-rpi-v8 #1  Debian 1:6.1.63-1+rpt1
[Sat Jan 20 18:56:58 2024] Hardware name: Raspberry Pi 4 Model B Rev 1.5 (DT)
[Sat Jan 20 18:56:58 2024] Call trace:
[Sat Jan 20 18:56:58 2024]  dump_backtrace.part.0+0xec/0x100
[Sat Jan 20 18:56:58 2024]  show_stack+0x20/0x30
[Sat Jan 20 18:56:58 2024]  dump_stack_lvl+0x88/0xb4
[Sat Jan 20 18:56:58 2024]  dump_stack+0x18/0x34
[Sat Jan 20 18:56:58 2024]  dump_header+0x4c/0x210
[Sat Jan 20 18:56:58 2024]  oom_kill_process+0x2bc/0x2c0
[Sat Jan 20 18:56:58 2024]  out_of_memory+0xe8/0x354
[Sat Jan 20 18:56:58 2024]  __alloc_pages+0xb04/0xea0
[Sat Jan 20 18:56:58 2024]  __folio_alloc+0x1c/0x30
[Sat Jan 20 18:56:58 2024]  __filemap_get_folio+0x1e0/0x460
[Sat Jan 20 18:56:58 2024]  filemap_fault+0x490/0x910
[Sat Jan 20 18:56:58 2024]  __do_fault+0x44/0x114
[Sat Jan 20 18:56:58 2024]  __handle_mm_fault+0x390/0xa40
[Sat Jan 20 18:56:58 2024]  handle_mm_fault+0x1b8/0x2d0
[Sat Jan 20 18:56:58 2024]  do_page_fault+0x194/0x3c4
[Sat Jan 20 18:56:58 2024]  do_translation_fault+0xa4/0xcc
[Sat Jan 20 18:56:58 2024]  do_mem_abort+0x4c/0xa0
[Sat Jan 20 18:56:58 2024]  el0_ia+0x74/0x10c
[Sat Jan 20 18:56:58 2024]  el0t_64_sync_handler+0xc4/0x120
[Sat Jan 20 18:56:58 2024]  el0t_64_sync+0x18c/0x190
[Sat Jan 20 18:56:58 2024] Mem-Info:
[Sat Jan 20 18:56:58 2024] active_anon:131107 inactive_anon:71424 isolated_anon:0
                            active_file:24 inactive_file:2 isolated_file:0
                            unevictable:0 dirty:1 writeback:63
                            slab_reclaimable:11665 slab_unreclaimable:5784
                            mapped:25 shmem:0 pagetables:884
                            sec_pagetables:0 bounce:0
                            kernel_misc_reclaimable:0
                            free:5037 free_pcp:0 free_cma:237
[Sat Jan 20 18:56:58 2024] Node 0 active_anon:343104kB inactive_anon:467020kB active_file:152kB inactive_file:152kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:100kB dirty:4kB writeback:252kB shmem:0kB writeback_tmp:0kB kernel_stack:2528kB pagetables:3536kB sec_pagetables:0kB all_unreclaimable? no
[Sat Jan 20 18:56:58 2024] DMA free:20148kB boost:4096kB min:20480kB low:24576kB high:28672kB reserved_highatomic:0KB active_anon:343104kB inactive_anon:467020kB active_file:0kB inactive_file:780kB unevictable:0kB writepending:256kB present:970752kB managed:930204kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:948kB
[Sat Jan 20 18:56:58 2024] lowmem_reserve[]: 0 0 0 0
[Sat Jan 20 18:56:58 2024] DMA: 1045*4kB (UMEC) 564*8kB (UMEC) 247*16kB (UMEC) 134*32kB (UMEC) 33*64kB (UME) 11*128kB (UMEC) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 20452kB
[Sat Jan 20 18:56:58 2024] 1421 total pagecache pages
[Sat Jan 20 18:56:58 2024] 1321 pages in swap cache
[Sat Jan 20 18:56:58 2024] Free swap  = 24kB
[Sat Jan 20 18:56:58 2024] Total swap = 102396kB
[Sat Jan 20 18:56:58 2024] 242688 pages RAM
[Sat Jan 20 18:56:58 2024] 0 pages HighMem/MovableOnly
[Sat Jan 20 18:56:58 2024] 10137 pages reserved
[Sat Jan 20 18:56:58 2024] 131072 pages cma reserved
[Sat Jan 20 18:56:58 2024] Tasks state (memory values in pages):
[Sat Jan 20 18:56:58 2024] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[Sat Jan 20 18:56:58 2024] [    247]     0   247    12481        0    98304      283          -250 systemd-journal
[Sat Jan 20 18:56:58 2024] [    265]     0   265     6525        0    65536      702         -1000 systemd-udevd
[Sat Jan 20 18:56:58 2024] [    371]   997   371    22645        0    73728      246             0 systemd-timesyn
[Sat Jan 20 18:56:58 2024] [    442]   104   442     1840        0    45056      115             0 avahi-daemon
[Sat Jan 20 18:56:58 2024] [    448]     0   448     1673        1    40960       59             0 cron
[Sat Jan 20 18:56:58 2024] [    449]   100   449     2172       59    49152      156          -900 dbus-daemon
[Sat Jan 20 18:56:58 2024] [    462]   996   462    58828        0    81920      264             0 polkitd
[Sat Jan 20 18:56:58 2024] [    465]   104   465     1795        0    45056       77             0 avahi-daemon
[Sat Jan 20 18:56:58 2024] [    481]     0   481     4256        0    65536      278             0 systemd-logind
[Sat Jan 20 18:56:58 2024] [    485] 65534   485     1457        0    40960       66             0 thd
[Sat Jan 20 18:56:58 2024] [    518]     0   518    65584      223   139264      560             0 NetworkManager
[Sat Jan 20 18:56:58 2024] [    529]     0   529     4288       24    65536      360             0 wpa_supplicant
[Sat Jan 20 18:56:58 2024] [    539]     0   539    80859        0   118784      501             0 ModemManager
[Sat Jan 20 18:56:58 2024] [    688]     0   688     3320        0    57344      153             0 bluetoothd
[Sat Jan 20 18:56:58 2024] [    689]     0   689     4082        0    65536      339         -1000 sshd
[Sat Jan 20 18:56:58 2024] [    741]     0   741     1405        0    40960       31             0 agetty
[Sat Jan 20 18:56:58 2024] [    834]     0   834     4865        0    73728      406             0 sshd
[Sat Jan 20 18:56:58 2024] [    837]  1000   837     4862        0    73728      438           100 systemd
[Sat Jan 20 18:56:58 2024] [    838]  1000   838    42827        0    98304     1124           100 (sd-pam)
[Sat Jan 20 18:56:58 2024] [    858]  1000   858     4967       37    77824      461             0 sshd
[Sat Jan 20 18:56:58 2024] [    943]  1000   943     2037       24    49152      401             0 bash
[Sat Jan 20 18:56:58 2024] [   1857]  1000  1857     2090        5    45056      465             0 bash
[Sat Jan 20 18:56:58 2024] [   1972]  1000  1972   297829   200768  1937408    16296             0 conda
[Sat Jan 20 18:56:58 2024] [   1973]  1000  1973     4253        1    69632     1236             0 python
[Sat Jan 20 18:56:58 2024] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=conda,pid=1972,uid=1000
[Sat Jan 20 18:56:58 2024] Out of memory: Killed process 1972 (conda) total-vm:1191316kB, anon-rss:803072kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:1892kB oom_score_adj:0

Installed packages

$ conda list
# packages in environment at /home/dtmcdonald/miniforge3:
#
# Name                    Version                   Build  Channel
_openmp_mutex             4.5                       2_gnu    conda-forge
archspec                  0.2.2              pyhd8ed1ab_0    conda-forge
boltons                   23.1.1             pyhd8ed1ab_0    conda-forge
brotli-python             1.1.0           py310hbb3657e_1    conda-forge
bzip2                     1.0.8                h31becfc_5    conda-forge
c-ares                    1.24.0               h31becfc_0    conda-forge
ca-certificates           2023.11.17           hcefe29a_0    conda-forge
certifi                   2023.11.17         pyhd8ed1ab_0    conda-forge
cffi                      1.16.0          py310hce94938_0    conda-forge
charset-normalizer        3.3.2              pyhd8ed1ab_0    conda-forge
colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
conda                     23.11.0         py310h4c7bcd0_1    conda-forge
conda-libmamba-solver     23.12.0            pyhd8ed1ab_0    conda-forge
conda-package-handling    2.2.0              pyh38be061_0    conda-forge
conda-package-streaming   0.9.0              pyhd8ed1ab_0    conda-forge
distro                    1.8.0              pyhd8ed1ab_0    conda-forge
fmt                       10.1.1               h2a328a1_1    conda-forge
icu                       73.2                 h787c7f5_0    conda-forge
idna                      3.6                pyhd8ed1ab_0    conda-forge
jsonpatch                 1.33               pyhd8ed1ab_0    conda-forge
jsonpointer               2.4             py310h4c7bcd0_3    conda-forge
keyutils                  1.6.1                h4e544f5_0    conda-forge
krb5                      1.21.2               hc419048_0    conda-forge
ld_impl_linux-aarch64     2.40                 h2d8c526_0    conda-forge
libarchive                3.7.2                hd2f85e0_1    conda-forge
libcurl                   8.5.0                h4e8248e_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 h31becfc_2    conda-forge
libffi                    3.4.2                h3557bc0_5    conda-forge
libgcc-ng                 13.2.0               hf8544c7_3    conda-forge
libgomp                   13.2.0               hf8544c7_3    conda-forge
libiconv                  1.17                 h31becfc_2    conda-forge
libmamba                  1.5.5                hea3be6c_0    conda-forge
libmambapy                1.5.5           py310h5938bc3_0    conda-forge
libnghttp2                1.58.0               hb0e430d_1    conda-forge
libnsl                    2.0.1                h31becfc_0    conda-forge
libsolv                   0.7.27               hd84c7bf_0    conda-forge
libsqlite                 3.44.2               h194ca79_0    conda-forge
libssh2                   1.11.0               h492db2e_0    conda-forge
libstdcxx-ng              13.2.0               h9a76618_3    conda-forge
libuuid                   2.38.1               hb4cce97_0    conda-forge
libxml2                   2.12.3               h3091e33_0    conda-forge
libzlib                   1.2.13               h31becfc_5    conda-forge
lz4-c                     1.9.4                hd600fc2_0    conda-forge
lzo                       2.10              h516909a_1000    conda-forge
mamba                     1.5.5           py310hcbdc16a_0    conda-forge
menuinst                  2.0.1           py310h4c7bcd0_0    conda-forge
ncurses                   6.4                  h0425590_2    conda-forge
openssl                   3.2.0                h31becfc_1    conda-forge
packaging                 23.2               pyhd8ed1ab_0    conda-forge
pip                       23.3.2             pyhd8ed1ab_0    conda-forge
platformdirs              4.1.0              pyhd8ed1ab_0    conda-forge
pluggy                    1.3.0              pyhd8ed1ab_0    conda-forge
pybind11-abi              4                    hd8ed1ab_3    conda-forge
pycosat                   0.6.6           py310hb299538_0    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
python                    3.10.13         hbbe8eec_0_cpython    conda-forge
python_abi                3.10                    4_cp310    conda-forge
readline                  8.2                  h8fc344f_1    conda-forge
reproc                    14.2.4.post0         h31becfc_1    conda-forge
reproc-cpp                14.2.4.post0         h2f0025b_1    conda-forge
requests                  2.31.0             pyhd8ed1ab_0    conda-forge
ruamel.yaml               0.18.5          py310hb299538_0    conda-forge
ruamel.yaml.clib          0.2.7           py310hb299538_2    conda-forge
setuptools                68.2.2             pyhd8ed1ab_0    conda-forge
tk                        8.6.13               h194ca79_0    conda-forge
tqdm                      4.66.1             pyhd8ed1ab_0    conda-forge
truststore                0.8.0              pyhd8ed1ab_0    conda-forge
tzdata                    2023c                h71feb2d_0    conda-forge
urllib3                   2.1.0              pyhd8ed1ab_0    conda-forge
wheel                     0.42.0             pyhd8ed1ab_0    conda-forge
xz                        5.2.6                h9cdd2b7_0    conda-forge
yaml-cpp                  0.8.0                h2f0025b_0    conda-forge
zstandard                 0.22.0          py310h468e293_0    conda-forge
zstd                      1.5.5                h4c53e97_0    conda-forge

Environment info

$ conda info

     active environment : base
    active env location : /home/dtmcdonald/miniforge3
            shell level : 1
       user config file : /home/dtmcdonald/.condarc
 populated config files : /home/dtmcdonald/miniforge3/.condarc
          conda version : 23.11.0
    conda-build version : not installed
         python version : 3.10.13.final.0
                 solver : libmamba (default)
       virtual packages : __archspec=1=aarch64
                          __conda=23.11.0=0
                          __glibc=2.36=0
                          __linux=6.1.0=0
                          __unix=0=0
       base environment : /home/dtmcdonald/miniforge3  (writable)
      conda av data dir : /home/dtmcdonald/miniforge3/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/linux-aarch64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : /home/dtmcdonald/miniforge3/pkgs
                          /home/dtmcdonald/.conda/pkgs
       envs directories : /home/dtmcdonald/miniforge3/envs
                          /home/dtmcdonald/.conda/envs
               platform : linux-aarch64
             user-agent : conda/23.11.0 requests/2.31.0 CPython/3.10.13 Linux/6.1.0-rpi7-rpi-v8 debian/12.4 glibc/2.36 solver/libmamba conda-libmamba-solver/23.12.0 libmambapy/1.5.5
                UID:GID : 1000:1000
             netrc file : None
           offline mode : False
hmaarrfk commented 5 months ago

1GB of ram is tough. Not sure if that is ever going to be possible in a meaningful timeframe for you.

Maybe consider using mamba or micromamba

wasade commented 5 months ago

Totally understand :) And not urgent, the quick workaround is bump swap. Note though this did work a month ago

hmaarrfk commented 5 months ago

The conda forge channel continues to grow. So the solver likely has a harder and harder time.