compilation exception when compiling in home directory

[lgo]

This is a bit tricky to provide reproducible steps, but this has happened in a school environment where Crystal was set up. As far as I know, it was set up using tar.gz.

The output of uname -a is:

Linux ubuntu1604-006 4.4.0-109-generic #132-Ubuntu SMP Tue Jan 9 19:52:39 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

The environment seems to have only LLVM 3.8 although Crystal seems to report 4.0.

$ crystal -v
Crystal 0.24.1 (2017-12-22)

LLVM: 4.0.0
Default target: x86_64-unknown-linux-gnu

If my current directory is my home directory, I was unable to compile anything with Crystal. I first tried using crystal eval and then a file in the home directory (hello.cr) and with crystal eval. Both inputs consisted of only puts "Hello". If I remain in $HOME and move the file to $HOME/tmp-crystal compilation succeeded but the first part of the error was printed (#<Exception:No such file or directory>...). Now, if I change the current directory to somewhere else, $HOME/tmp-crystal, compilation seems to succeed with no errors whether or not the file was in $HOME.

Those were the exact steps that I had done. Now the peculiar part is that once I had fiddled around, compiling in different directories it always seems to succeed if I'm compiling a file. If I am in $HOME the first part of the error is still printed. But if I compile using crystal eval in my home directory, the same error from the beginning still happens and compilation fails.

The output of the error is the following (the "beginning" refers to the first line with #<Exception:No such file or directory> ...):

$ crystal build hello.cr
#<Exception:No such file or directory>#<Exception:No such file or directory>#<Exception:No such file or directory>#<Exception:No such file or directory>#<Exception:No such file or directory>#<Exception:No such file or directory>#<Exception:No such file or directory>#<Exception:No such file or directory>#<Exception:No such file or directory>cc: error: _main.o: No such file or directory
cc: error: S-lice40U-I-nt841.o: No such file or directory
cc: error: P-ointer40U-I-nt841.o: No such file or directory
cc: error: A-rgumentE-rror.o: No such file or directory
cc: error: C-allS-tack.o: No such file or directory
cc: error: A-rray40P-ointer40V-oid4141.o: No such file or directory
cc: error: P-ointer40P-ointer40V-oid4141.o: No such file or directory
cc: error: E-xception.o: No such file or directory
cc: error: P-ointer40L-ibU-nwind5858E-xception41.o: No such file or directory
cc: error: U-I-nt64.o: No such file or directory
cc: error: U-I-nt8.o: No such file or directory
cc: error: D-ir.o: No such file or directory
cc: error: S-tring.o: No such file or directory
cc: error: G-C-.o: No such file or directory
cc: error: E-rrno.o: No such file or directory
cc: error: S-tring5858B-uilder.o: No such file or directory
cc: error: S-lice40T-41.o: No such file or directory
cc: error: I-O-5858E-ncoder.o: No such file or directory
cc: error: I-conv.o: No such file or directory
cc: error: M-ath.o: No such file or directory
cc: error: I-nt32.o: No such file or directory
cc: error: S-taticA-rray40U-I-nt84432102441.o: No such file or directory
cc: error: I-ndexE-rror.o: No such file or directory
cc: error: C-har.o: No such file or directory
cc: error: S-taticA-rray40U-I-nt84432441.o: No such file or directory
cc: error: I-nvalidB-yteS-equenceE-rror.o: No such file or directory
cc: error: S-taticA-rray40U-I-nt8443212941.o: No such file or directory
cc: error: D-ivisionB-yZ-ero.o: No such file or directory
cc: error: A-rray40S-tring41.o: No such file or directory
cc: error: P-ointer40S-tring41.o: No such file or directory
cc: error: S-taticA-rray40I-nt8443225641.o: No such file or directory
cc: error: P-ointer40I-nt841.o: No such file or directory
cc: error: P-ointer40V-oid41.o: No such file or directory
cc: error: F-iber.o: No such file or directory
cc: error: T-hread.o: No such file or directory
cc: error: S-et40T-hread41.o: No such file or directory
cc: error: H-ash40T-hread4432N-il41.o: No such file or directory
cc: error: P-ointer40H-ash58-5c8fd5776cf345ec94b91bf7a5f7e50f.o: No such file or directory
cc: error: A-rray40I-nt3241.o: No such file or directory
cc: error: P-ointer40I-nt3241.o: No such file or directory
cc: error: C-rystal5858H-asher.o: No such file or directory
cc: error: S-taticA-rray40U-I-nt644432241.o: No such file or directory
cc: error: P-ointer40U-I-nt6441.o: No such file or directory
cc: error: H-ash5858E-ntry40T-hread4432N-il41.o: No such file or directory
cc: error: N-il.o: No such file or directory
cc: error: D-eque40F-iber41.o: No such file or directory
cc: error: P-ointer40F-iber41.o: No such file or directory
cc: error: E-vent5858B-ase.o: No such file or directory
cc: error: R-andom5858S-ecure.o: No such file or directory
cc: error: C-rystal5858S-ystem5858R-andom.o: No such file or directory
cc: error: T-ime5858S-pan.o: No such file or directory
cc: error: I-nt64.o: No such file or directory
cc: error: S-cheduler.o: No such file or directory
cc: error: E-vent5858E-vent.o: No such file or directory
cc: error: L-ibC-5858T-imeval.o: No such file or directory
cc: error: I-O-5858F-ileD-escriptor.o: No such file or directory
cc: error: I-O-5858F-ileD-escriptor43.o: No such file or directory
cc: error: I-O-5858E-rror.o: No such file or directory
cc: error: L-ibE-vent25858E-ventF-lags.o: No such file or directory
cc: error: I-O-5858T-imeout.o: No such file or directory
cc: error: R-ange40B-4432E-41.o: No such file or directory
cc: error: R-ange40I-nt324432I-nt3241.o: No such file or directory
cc: error: E-xception43.o: No such file or directory
cc: error: E-N-V-.o: No such file or directory
cc: error: F-ile.o: No such file or directory
cc: error: C-har5858R-eader.o: No such file or directory
cc: error: U-I-nt32.o: No such file or directory
cc: error: D-ebug5858E-L-F-.o: No such file or directory
cc: error: D-ebug5858E-L-F-5858E-rror.o: No such file or directory
cc: error: D-ebug5858E-L-F-5858E-ndianness.o: No such file or directory
cc: error: D-ebug5858E-L-F-5858O-S-A-B-I-.o: No such file or directory
cc: error: S-taticA-rray40U-I-nt84432409641.o: No such file or directory
cc: error: I-O-5858E-O-F-E-rror.o: No such file or directory
cc: error: D-ebug5858E-L-F-5858I-dent.o: No such file or directory
cc: error: U-I-nt16.o: No such file or directory
cc: error: I-O-5858B-yteF-ormat5858B-igE-ndian.o: No such file or directory
cc: error: S-taticA-rray40U-I-nt84432241.o: No such file or directory
cc: error: I-O-5858B-yteF-ormat5858L-ittleE-ndian.o: No such file or directory
cc: error: D-ebug5858E-L-F-5858K-lass.o: No such file or directory
cc: error: S-taticA-rray40U-I-nt84432841.o: No such file or directory
cc: error: A-rray40D-ebug5858E-L-F-5858S-ectionH-eader41.o: No such file or directory
cc: error: P-ointer40D-ebug5858E-L-F-5858S-ectionH-eader41.o: No such file or directory
cc: error: I-O-5858S-eek.o: No such file or directory
cc: error: D-ebug5858E-L-F-5858S-ectionH-eader.o: No such file or directory
cc: error: I-O-5858D-ecoder.o: No such file or directory
cc: error: D-ebug5858D-W-A-R-F-5858L-ineN-umbers.o: No such file or directory
cc: error: A-rray40A-rray40D-d5580c0e9cb0bd8af7e90c96221706e1.o: No such file or directory
cc: error: P-ointer40A-rray4-6533bdad81433bcf155b99ac74756686.o: No such file or directory
cc: error: D-ebug5858D-W-A-R-F-5858L-ineN-umbers5858S-equence.o: No such file or directory
cc: error: A-rray40T-uple40S-5992430a1a7ef1627717adfb71332538.o: No such file or directory
cc: error: P-ointer40T-uple4-13415771527eed75ea796bf4f3711255.o: No such file or directory
cc: error: A-rray40U-I-nt841.o: No such file or directory
cc: error: I-nt8.o: No such file or directory
cc: error: S-taticA-rray40U-I-nt84432141.o: No such file or directory
cc: error: D-ebug5858D-W-A-R-F-.o: No such file or directory
cc: error: D-ebug5858D-W-A-R-F-5858L-ineN-umbers5858R-egister.o: No such file or directory
cc: error: D-ebug5858D-W-A-R-F-5858L-ineN-umbers5858R-ow.o: No such file or directory
cc: error: A-rray40D-ebug585-1d195a7f864d2d3790ebeadd84b7a4df.o: No such file or directory
cc: error: P-ointer40D-ebug5-a50eac5c43900f6bca76f68ff8c8549d.o: No such file or directory
cc: error: D-ebug5858D-W-A-R-F-5858L-N-E-.o: No such file or directory
cc: error: D-ebug5858D-W-A-R-F-5858L-N-S-.o: No such file or directory
cc: error: D-ebug5858D-W-A-R-F-5858S-trings.o: No such file or directory
cc: error: A-rray40T-uple40U-997f9e533f193da5e605a2014e7947d9.o: No such file or directory
cc: error: P-ointer40T-uple4-aa2454898accd295538d997a0365d41d.o: No such file or directory
cc: error: D-ebug5858D-W-A-R-F-5858I-nfo.o: No such file or directory
cc: error: D-ebug5858D-W-A-R-F-5858A-bbrev.o: No such file or directory
cc: error: A-rray40D-ebug5858D-W-A-R-F-5858A-bbrev41.o: No such file or directory
cc: error: P-ointer40D-ebug5858D-W-A-R-F-5858A-bbrev41.o: No such file or directory
cc: error: A-rray40D-ebug585-b448de41619185732a02b2b0c778beb9.o: No such file or directory
cc: error: P-ointer40D-ebug5-038cd8f166075bfacc819464bf65667a.o: No such file or directory
cc: error: D-ebug5858D-W-A-R-F-5858A-bbrev5858A-ttribute.o: No such file or directory
cc: error: A-rray40T-uple40D-2934b2835f929d23ac3adb7e5b7b0afe.o: No such file or directory
cc: error: P-ointer40T-uple4-6ff119ef58145d1144587389190a4e25.o: No such file or directory
cc: error: D-ebug5858D-W-A-R-F-5858F-O-R-M-.o: No such file or directory
cc: error: U-nicode.o: No such file or directory
cc: error: U-nicode5858C-aseO-ptions.o: No such file or directory
cc: error: A-rray40T-uple40I-nt324432I-nt324432I-nt324141.o: No such file or directory
cc: error: P-ointer40T-uple40I-nt324432I-nt324432I-nt324141.o: No such file or directory
cc: error: H-ash40I-nt324432-c36c7b59ada9577e04b34b7156d26117.o: No such file or directory
cc: error: P-ointer40H-ash58-20170b0dcbfcdda132d3c3911ed0b4b4.o: No such file or directory
cc: error: H-ash5858E-ntry40-ea883736b6a497e8c0f2d29083b2162a.o: No such file or directory
cc: error: A-rray40T-uple40I-nt324432I-nt324141.o: No such file or directory
cc: error: P-ointer40T-uple40I-nt324432I-nt324141.o: No such file or directory
cc: error: D-ebug5858D-W-A-R-F-5858T-A-G-.o: No such file or directory
cc: error: D-ebug5858D-W-A-R-F-5858A-T-.o: No such file or directory
cc: error: B-ool.o: No such file or directory
cc: error: T-ypeC-astE-rror.o: No such file or directory
cc: error: S-tring5858T-oU-64I-nfo.o: No such file or directory
cc: error: P-roc40F-iber4432-6eb246a0a45118d3c5507cc830b14a70.o: No such file or directory
cc: error: F-ile5858S-tat.o: No such file or directory
cc: error: S-ignal.o: No such file or directory
cc: error: P-roc40I-nt324432V-oid41.o: No such file or directory
cc: error: E-vent5858S-ignalH-andler.o: No such file or directory
cc: error: H-ash40S-ignal4432P-roc40S-ignal4432N-il4141.o: No such file or directory
cc: error: P-ointer40H-ash58-ac8d12c9042874d67c1e049ddde35350.o: No such file or directory
cc: error: H-ash5858E-ntry40-37a9b13947932e73bc9389d755897071.o: No such file or directory
cc: error: E-vent5858S-ignalC-hildH-andler.o: No such file or directory
cc: error: H-ash40I-nt324432P-rocess5858S-tatus41.o: No such file or directory
cc: error: P-ointer40H-ash58-0a6eee72af0923261bdd3b6c7958da85.o: No such file or directory
cc: error: H-ash40I-nt324432-12de0ef574784dde6ffe5b3ae93d5bba.o: No such file or directory
cc: error: P-ointer40H-ash58-1ce4187043ca1e746f24acb8d0ff7a7e.o: No such file or directory
cc: error: P-rocess5858S-tatus.o: No such file or directory
cc: error: C-hannel5858B-uff-35976990a41c6a8201ab74e37227bdef.o: No such file or directory
cc: error: C-hannel5858C-losedE-rror.o: No such file or directory
cc: error: D-eque40P-rocess5858S-tatus3212432N-il41.o: No such file or directory
cc: error: P-ointer40P-rocess5858S-tatus3212432N-il41.o: No such file or directory
cc: error: H-ash5858E-ntry40-8e9a3cfcf19f0b3833711caadc35e784.o: No such file or directory
cc: error: H-ash5858E-ntry40I-nt324432P-rocess5858S-tatus41.o: No such file or directory
cc: error: I-O-.o: No such file or directory
cc: error: S-taticA-rray40I-nt324432241.o: No such file or directory
cc: error: A-tE-xitH-andlers.o: No such file or directory
cc: error: A-rray40P-roc40I-nt324432N-il4141.o: No such file or directory
cc: error: P-ointer40P-roc40I-nt324432N-il4141.o: No such file or directory
cc: error: L-E-B-R-eader.o: No such file or directory
cc: error: L-ibU-nwind5858A-ction.o: No such file or directory
cc: error: L-ibU-nwind5858R-easonC-ode.o: No such file or directory
cc: error: C-allS-tack5858R-epeatedF-rame.o: No such file or directory
cc: error: C-rystal.o: No such file or directory
Error: execution of command failed with code: 1: `cc "${@}" -o '/u3/j5pereir/hello'  -rdynamic  -lpcre -lgc -lpthread /usr/share/crystal/src/ext/libcrystal.a -levent -lrt -ldl -L/usr/lib -L/usr/local/lib`

[straight-shoota]

This could be difficult to reproduce without any further details about the environment. What exactly is your home path (echo $HOME)?

Out of curiosity, which school is using Crystal and for what? =)

[lgo]

(oops, I thought I put it in the original comment).

$ echo $HOME
/u3/j5pereira

The school isn't quite using it, but it is the University of Waterloo. This is for a compilers course where we can use any language we want, so they made it available at my request. =)

[RX14]

@lego can you run crystal env please?

[drosehn]

It might also be interesting to see the output of echo $PATH

[lgo]

$ crystal env
CRYSTAL_CACHE_DIR="/u3/j5pereira/.cache/crystal"
CRYSTAL_PATH="/usr/bin/../share/crystal/src:lib"
CRYSTAL_VERSION="0.24.1"

$ echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games

The cache folder stood out so I poked around it. It appears that compilation fails while the current directory is the home directory and if there is no cache populated for the targeted crystal build file (regardless of where the target crystal file is). By removing the cache I'm able to reproduce the build problem. The problem no longer happens after doing a single build while the current directory is somewhere else (i.e. $HOME/tmp-crystal). After that, the cache folder for the build target has all of the files that are reported as missing.

Something else that stood out is that /usr/bin/../share/crystal only contains the folder src. Should it also have lib?

A few other diagnostics:

The permissions on $HOME/.cache and $HOME/.cache/crystal

drwx--S---   41 j5pereir none         4096 Jan 30 13:59 .cache
drwxr-sr-x  3 j5pereir none       4096 Jan 30 13:59 crystal

The contents of $HOME/.cache/crystal/u3-j5pereir-hello.cr after a failed compilation.

-rw-r--r-- 1 j5pereir none  3424 Jan 30 14:17 A-rray40I-nt3241.bc
-rw-r--r-- 1 j5pereir none     9 Jan 30 14:17 bc_flags
-rw-r--r-- 1 j5pereir none  5020 Jan 30 14:17 D-ebug5858D-W-A-R-F-5858L-ineN-umbers5858R-egister.bc
-rw-r--r-- 1 j5pereir none 11116 Jan 30 14:17 H-ash40S-ignal4432P-roc40S-ignal4432N-il4141.bc
-rw-r--r-- 1 j5pereir none  2156 Jan 30 14:17 I-O-5858E-rror.bc
-rw-r--r-- 1 j5pereir none 56280 Jan 30 14:17 _main.bc
-rw-r--r-- 1 j5pereir none  2236 Jan 30 14:17 M-ath.bc
-rw-r--r-- 1 j5pereir none  3116 Jan 30 14:17 P-ointer40P-roc40I-nt324432N-il4141.bc
-rw-r--r-- 1 j5pereir none  2508 Jan 30 14:17 S-taticA-rray40U-I-nt84432241.bc
-rw-r--r-- 1 j5pereir none 45212 Jan 30 14:17 U-nicode.bc

[lgo]

I also ran strace crystal build $HOME/hello.cr immediately after deleting .cache/crystal. I haven't yet groked the results. Hopefully, this may give some indication as to what the binary is looking for and why it's failing. crystal-build-strace.log. I don't know if I'll come back to the output until later this week so if someone knows more about what's going on feel free to go through it :)

[jhass]

@lgo since it's been over two years, do you mind trying again? :)

[tuxor1337]

I'm able to reproduce a similar error output, but I'm not sure if it is related. But it might well be, because it's also on a shared host. The background is that I want to compile the crystal project https://github.com/omarroth/invidious as a normal user on a shared host (via ssh). When running the crystal-build command with --threads 1, the job is killed at some point because the shared host has restrictions on resource use by normal users. However, I can simply rerun the build command several times until, at some point, it will succeed.

To reproduce the error output in this issue, I have to omit the argument --threads 1. I suspect that the host is killing compiler threads and the crystal builder misinterprets this as finished threads. It invokes the linking steps before the compiler is actually finished leading to missing object files. If this is how crystal-build interprets killed threads, this might also explain related issues such as https://github.com/crystal-lang/crystal/issues/7993

Details: I install all the dependencies into a conda environment like here https://tovotu.de/dev/524-Invidious-auf-einem-Uberspace-einrichten/ (unfortunately, only in German).

My setup:

$ uname -a
Linux cordelia.uberspace.de 3.10.0-1127.10.1.el7.x86_64 #1 SMP Wed Jun 3 14:28:03 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
$ crystal version
Crystal 0.35.0 [3c48f311f] (2020-06-09)

LLVM: 8.0.0
Default target: x86_64-unknown-linux-gnu
$ crystal env 
CRYSTAL_CACHE_DIR=/home/user8297/.cache/crystal
CRYSTAL_PATH=lib:/home/user8297/opt/crystal/bin/../share/crystal/src
CRYSTAL_VERSION=0.35.0
CRYSTAL_LIBRARY_PATH=/home/user8297/opt/crystal/bin/../lib/crystal/lib
CRYSTAL_OPTS=''
$ $CC --version
x86_64-conda_cos6-linux-gnu-cc (crosstool-NG 1.23.0.449-a04d0) 7.3.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

[bbugh]

I am able to reproduce this with 100% consistency while trying to set up mint on the crystal 1.0.0 docker image. It is not in the home directory, but is in ~/mint.

$ docker run -t -i crystallang/crystal:1.0.0 /bin/bash
$ cd ~
$ git clone https://github.com/mint-lang/mint
$ cd mint
$ shards install
$ apt update
$ apt install pkg-config ubuntu-dev-tools
$ mkdir ~/.bin
$ make development

It will then die:

root@0cabd58b0506:~/mint# make development
shards build --error-on-warnings --error-trace --progress
Dependencies are satisfied
Building: mint
Error target mint failed to compile:
cc: error: _main.o: No such file or directory
cc: error: S-lice40U-I-nt841.o: No such file or directory
cc: error: P-ointer40U-I-nt841.o: No such file or directory
cc: error: A-rgumentE-rror.o: No such file or directory
cc: error: E-xception5858C-allS-tack.o: No such file or directory
cc: error: A-rray40P-ointer40V-oid4141.o: No such file or directory
cc: error: P-ointer40P-ointer40V-oid4141.o: No such file or directory
cc: error: I-nt32.o: No such file or directory
cc: error: F-loat64.o: No such file or directory

[straight-shoota]

I am unable to reproduce with these exact same steps.

[bbugh]

Isn't coding wonderful? 😂 I'm on a week old Mac Big Sur 11.3, if that matters. To debug further:

1. I tried using `ubuntu:20.04" (adf73ca0) image instead of the crystal one, and installed everything manually, and it still had the same problem.
2. I did a full boot to a Linux Mint live usb (v20.2) and it worked fine
3. I booted back to the Mac and tried running the same steps I posted above with the linuxmintd/mint20.2-amd64 image instead. It also failed in the same spot.
4. I finally tried crystallang/crystal:nightly-alpine-build (0f7866bb39f0) with the equivalent steps and it worked without any issues. ✅

It seems like it might be a debian thing?

[straight-shoota]

I would suspect it has something to do with the file system. Or the underlying disc device.

[mloughran]

I landed here while debugging a similar issue. Builds (for certain targets – presumably linked to compilation complexity) were failing within a docker container limited to 2GB RAM. The solution was to increase Docker's RAM allowance. If the root cause here is indeed OOM, it would be nice if the error could be more helpful :)

[straight-shoota]

Indeed, I have recently encountered this as well and I'm pretty sure it was because there was not enough memory available.

We should probably investigate why this particular behaviour happens on low memory. It seems to be a rather odd symptom for that. And then of course improve the compiler's error reporting.

[atlantis]

Ooof I'm running into this sporadically trying to build on an arm64 IoT device (embedded crystal ftw)... I only have 2GB to work with on the device so cranking up the memory limit isn't an option unfortunately...

[straight-shoota]

@atlantis Unfortunately, the Crystal compiler is quite memory hungry, so it doesn't run well in confined environments such as embedded devices. Your best chance is probably cross compiling from a more potent host system.

[atlantis]

Yup! I'm using Balena and the production builds do that, but it's also nice to be able to compile on the device in devmode... crystal works surprisingly well overall on a constrained device.