kit-ty-kate
commented
1 year ago Could you give us the output the df -ih ?
Open jteg68 opened 1 year ago
kit-ty-kate
commented
1 year ago Could you give us the output the df -ih ?
jteg68
commented
1 year ago teg@hilbert:~$ df -ih Filesystem Inodes IUsed IFree IUse% Mounted on udev 971K 511 971K 1% /dev tmpfs 981K 1.1K 980K 1% /run /dev/sda2 1.5M 373K 1.1M 25% / tmpfs 981K 1 981K 1% /dev/shm tmpfs 981K 5 981K 1% /run/lock /dev/sda3 597K 19K 579K 4% /var /dev/sda5 120K 23K 97K 20% /tmp /dev/sda1 0 0 0 - /boot/efi /dev/sda6 115M 260K 114M 1% /home tmpfs 197K 131 196K 1% /run/user/1001 tmpfs 197K 142 196K 1% /run/user/1000
I copied the downloaded index.tar.gz from the /tmp/opam-xxx-xxx directory and extracted in a fresh directory under /tmp. No problem detected, the extracted files looks ok.
kit-ty-kate
commented
1 year ago /dev/sda5 120K 23K 97K 20% /tmp
mmh 97K inodes free might be on the low side for the opam-repository
Testing locally, after untaring https://opam.ocaml.org/index.tar.gz I've already used 61K.
tmpfs 992K 61K 931K 7% /tmp/testCould you try to see if an increase of the number of inode fixes your problem? sudo mount -t tmpfs tmpfs /tmp should give you a bit more (you can unmount it after and get back the files that were there before)
dbuenzli
commented
1 year ago Btw. just thinking out loud. Would it maybe make sense for opam to just work on the "file system" of the index.tar file ?
jteg68
commented
1 year ago Thanks, with more inodes the update sailed through. Working directly on the index.tar file, is there a cmd line option for that?
dbuenzli
commented
1 year ago @jteg68 no that was just me thinking very loudly.
kit-ty-kate
commented
1 year ago Thanks, with more inodes the update sailed through.
Awesome, good to know.
Looking at the difference between 2.1 and master, it seems like your issue should be fixed by using master:
git clone https://github.com/ocaml/opam
make -C opam cold
sudo install ./opam/opam /usr/local/bin/opamin master we untar directly into the opam root (~/.opam in your case) so /tmp isn't used as much.
Would it maybe make sense for opam to just work on the "file system" of the index.tar file ?
Mmmh, that's interesting, we could use tar -d but sadly this option seem to be a bit finicky and not portable (only present in GNU tar). Were you thinking of something else?
dbuenzli
commented
1 year ago Were you thinking of something else?
I have no idea how opam uses all the info so take the following with a bag of salt.
But I mean I don't know if it's useful to have all these files untared on my disk. Just mmap the tar file and navigate the tar format.
MdeLv
commented
1 year ago Hi
+1
Still don't know how to fix that.
One of my debian11/opam 2.1.4 server works well with opam --update.
Another one is stuck for a few days with the same message as @jteg68 got.
Doing opam init --reinit -ni does the same, or reinstalling opam triggers the same error:
$ bash -c "sh <(curl -fsSL https://raw.githubusercontent.com/ocaml/opam/master/shell/install.sh)"
## Downloading opam 2.1.4 for linux on x86_64...
## Downloaded.
## Where should it be installed ? [/usr/local/bin]
Write access to /usr/local/bin required, using 'sudo'.
Command: rm -f /usr/local/bin/opam
Write access to /usr/local/bin required, using 'sudo'.
Command: install -m 755 /tmp/opam-2.1.4-x86_64-linux /usr/local/bin/opam
## opam 2.1.4 installed to /usr/local/bin
## Converting the opam root format & updating
No configuration file found, using built-in defaults.
Checking for available remotes: rsync and local, git.
- you won't be able to use mercurial repositories unless you install the hg command on your system.
- you won't be able to use darcs repositories unless you install the darcs command on your system.
<><> Updating repositories ><><><><><><><><><><><><><><><><><><><><><><><><><><>
[multicore] no changes from git+https://github.com/ocamllabs/multicore-opam.git
[ERROR] Could not update repository "default": Failed to extract archive /tmp/opam-3628659-e3393d/index.tar.gz: "/usr/bin/tar xfz /tmp/opam-3628659-e3393d/index.tar.gz -C /tmp/opam-3628659-bfd99e/default.new"
exited with code 2I tried with 4 different so-called "switches" and got the same result.
I can manually get and extract the opam index:
wget https://opam.ocaml.org/index.tar.gz && tar xfz index.tar.gz -C tmp_opam/
btw, I don't know why a simple opam update takes so much time?
There is enough storage on /tmp and /home (GB).
What information do you need to know exactly what is going on with this opam configuration? I hope we are only a few people impacted by this trouble. Thanks.
ETA:
I remembered I had a "no space left on device" error on a low resources debian server.
I had to find where are opam logs to investigate that possibility (btw I found no information about log location in opam helpand not even the word "log" or each page of eh official documentation at https://opam.ocaml.org/doc/)
$ ll -h -tr ~/.opam/log/
-rw-r--r-- 1 ml ml 309 Mar 17 12:15 log-3628842-be11d0.info
-rw-r--r-- 1 ml ml 4.2K Mar 17 12:15 log-3628842-be11d0.env
-rw-r--r-- 1 ml ml 7.2M Mar 17 12:15 log-3628842-be11d0.outThere is space available on device ((opam used to use /tmp and uses /home):
df -h
Size Used Avail Use% Mounted on
7.8G 4.0K 7.8G 1% /dev
1.6G 3.5M 1.6G 1% /run
23G 20G 2.3G 90% /
1.8G 184M 1.6G 11% /tmp
9.1G 5.4G 3.3G 63% /var
304G 249G 41G 86% /home
...However, I'm surprised to see the following logs:
$ tail ~/.opam/log/log-3628842-be11d0.out
/usr/bin/tar: packages/nlopt-ocaml: Cannot mkdir: No space left on device
/usr/bin/tar: packages/nlopt-ocaml/nlopt-ocaml.0.4: Cannot mkdir: No such file or directory
/usr/bin/tar: packages/nlopt-ocaml: Cannot mkdir: No space left on device
/usr/bin/tar: packages/nlopt-ocaml/nlopt-ocaml.0.4/opam: Cannot open: No such file or directory
/usr/bin/tar: packages/nlopt-ocaml: Cannot mkdir: No space left on device
/usr/bin/tar: packages/nlopt-ocaml/nlopt-ocaml.0.5.1: Cannot mkdir: No such file or directory
/usr/bin/tar: packages/nlopt-ocaml: Cannot mkdir: No space left on device
/usr/bin/tar: packages/nlopt-ocaml/nlopt-ocaml.0.5.1/opam: Cannot open: No such file or directory
/usr/bin/tar: repo: Cannot open: No space left on device
/usr/bin/tar: Exiting with failure status due to previous errorsAs /tmp and /home appear to have enough space available, I can't tell what happens. Maybe just a tar error related to a non existent dir (see -C option used with tar)?...
btw, why is this precious as well as hidden log information not displayed when using opam update with --verbose?
kit-ty-kate
commented
1 year ago @MdeLv sorry you're experiencing the same issue. Please read the discussion above, the solution(s) and cause of the issue (lack of inodes, not lack of space) are all there already. If you want a summary, the solutions are either:
/tmp: sudo mount -t tmpfs tmpfs /tmp should fix ityou can install opam master with these 3 simple commands:
git clone https://github.com/ocaml/opam
make -C opam cold
sudo install ./opam/opam /usr/local/bin/opamThanks for your quick answer.
Solution 1 worked for me.
$ df -ih
Inodes IUsed IFree IUse% Mounted on
120K 43K 77K 36% /tmp
...sudo mount -t tmpfs tmpfs /tmp
$ df -ih
Inodes IUsed IFree IUse% Mounted on
2.0M 1 2.0M 1% /tmp
...Reinstalling opam the curl way (to override the installed binary I just compiled):
bash -c "sh <(curl -fsSL https://raw.githubusercontent.com/ocaml/opam/master/shell/install.sh)"
...
<><> Updating repositories ><><><><><><><><><><><><><><><><><><><><><><><><><><>
[default] synchronised from https://opam.ocaml.org
[multicore] no changes from git+https://github.com/ocamllabs/multicore-opam.gitNow OK! (same wih opam update)
Thanks.
btw, just before your message, I was trying the solution 2 (git clone https://github.com/ocaml/opam && make -C opam cold && sudo install ./opam/opam /usr/local/bin/opam).
As it is updating the repo from 2.1 to 2.2 alpha, it is asked to confirm that non reversible action, without proposing a backup.
What is the recommended way to backup an opam repository on the host?
Is it enough to consider ~/.opam and to make an archive from it?
MdeLv
commented
1 year ago !!! Breaking opam commands !!!
I initially had a side-comment about doing a backup of the opam repository configuration of the host before upgrading from 2.1 to 2.2 alpha.
I can see 107 GB in ~/.opam for 10 compilers and package sets, which seems huge to me.
opam switch list triggered an unusual message:
$ opam switch list
[ERROR] Opam has not been initialised, please run `opam init'Then:
$ opam init
No configuration file found, using built-in defaults.
Checking for available remotes: rsync and local, git.
- you won't be able to use mercurial repositories unless you install the hg
command on your system.
- you won't be able to use darcs repositories unless you install the darcs
command on your system.
...
<><> Creating initial switch 'default' (invariant ["ocaml" {>= "4.05.0"}] - initially with ocaml-base-compiler) During this compilation, I discovered that my compiler opam configurations disappeared. See details: https://github.com/ocaml/opam/issues/5485
rjbou
commented
1 year ago But I mean I don't know if it's useful to have all these files untared on my disk. Just mmap the tar file and navigate the tar format.
There is an issue about using tar library instead of the command, with that it would be possible to read loaded files without writing them.
jteg68
commented
1 year ago Hi again @kit-ty-kate ,
I just tried upgrading opam to master but the same problem persists. Was the change to use opam root reverted? Trying with a hard coded path in opamRepositoryState.ml, around line 173, the opam update works without issue.
jruere
commented
9 months ago Changing the temp path worked for me: TMPDIR=~/tmp/ opam update.
Hi, an 'opam update' gives an error 2. Any ideas on how to investigate this?
The man pages for tar (GNU tar 1.34) gives this for return code 2
The file system have a lot of room: