neurosymbolicgroup / neurosymbolic-modules

11 stars 6 forks source link

cannot build ocaml libs #1

Closed Seondong closed 2 years ago

Seondong commented 2 years ago

Is DreamCoder only run on the recent GLIBC version? (2.28 or 2.29?) The compression part seems to require GLIBC_2.28 or above. Most forum articles do not suggest updating glibc, so I would like to maintain the environment. Is this compatibility issue stemming from ocaml? Any workaround can you suggest?

Setup

In our setup, we observed the GLIBC version issue as below.

$ singularity exec container.img python -u bin/official_arc_symmetry.py -t 10 -g -i 2
singularity exec container.img python -u bin/official_arc_symmetry.py -t 10 -g -i 2
CUDA is available?: False
using cuda?: False
Running EC on node03 @ 2022-04-28 23:22:10.069179 with 1 CPUs and parameters:
         noConsolidation  =  False
         iterations  =  2
         enumerationTimeout  =  10
         useRecognitionModel  =  False
         topk_use_only_likelihood  =  False
         pseudoCounts  =  30.0
         aic  =  0.1
         structurePenalty  =  0.1
         arity  =  3
         taskReranker  =  default
         storeTaskMetrics  =  True
         rewriteTaskMetrics  =  False
         maximumFrontier  =  10
         solver  =  python
         topK  =  5
         evaluationTimeout  =  1.0
         cuda  =  False

Currently using this much memory: 139558912
Currently using this much memory: 139558912
Using a waking task batch of size: 36
Disabling parallelism on the Python side because we only have one job.
If you are using ocaml or bottom, there could still be parallelism.
(frontend) Launching input -> grid (36 tasks) w/ 1 CPUs. 0.000000 <= MDL < 1.500000. Timeout 10.000000.
(frontend) Launching input -> grid (36 tasks) w/ 1 CPUs. 1.500000 <= MDL < 3.000000. Timeout 9.884336.
(frontend) Launching input -> grid (36 tasks) w/ 1 CPUs. 3.000000 <= MDL < 4.500000. Timeout 9.748159.
(frontend) Launching input -> grid (36 tasks) w/ 1 CPUs. 4.500000 <= MDL < 6.000000. Timeout 9.627584.
(frontend) Launching input -> grid (36 tasks) w/ 1 CPUs. 6.000000 <= MDL < 7.500000. Timeout 9.466882.
(frontend) Launching input -> grid (36 tasks) w/ 1 CPUs. 7.500000 <= MDL < 9.000000. Timeout 9.331560.
(frontend) Launching input -> grid (35 tasks) w/ 1 CPUs. 9.000000 <= MDL < 10.500000. Timeout 8.932637.
(frontend) Launching input -> grid (30 tasks) w/ 1 CPUs. 10.500000 <= MDL < 12.000000. Timeout 7.100658.
We enumerated this many programs, for each task:
         [567, 2472, 2472, 2472, 2472, 2472, 2472, 567, 2472, 2472, 2472, 2472, 2472, 2472, 2472, 567, 2472, 111, 2472, 2472, 2472, 567, 2472, 2472, 2472, 567, 2472, 2472, 2472, 2472, 2472, 2472, 2472, 2472, 2472, 2472]
Generative model enumeration results:
HIT 30 w/ (lambda (object (input $0))) ; log prior = -3.891820 ; log likelihood = 0.000000
HIT 38 w/ (lambda (left_half (rotate_cw (left_half (object (input $0)))))) ; log prior = -9.729551 ; log likelihood = 0.000000
MISS 52
MISS 56
MISS 66
MISS 70
MISS 82
HIT 86 w/ (lambda (rotate_cw (rotate_cw (input $0)))) ; log prior = -5.837730 ; log likelihood = 0.000000
MISS 105
MISS 108
HIT 112 w/ (lambda (overlay (input $0) (x_mirror (input $0)))) ; log prior = -7.783641 ; log likelihood = 0.000000
HIT 115 w/ (lambda (combine_grids_vertically (x_mirror (input $0)) (input $0))) ; log prior = -7.783641 ; log likelihood = 0.000000
MISS 116
HIT 139 w/ (lambda (rotate_cw (rotate_cw (input $0)))) ; log prior = -5.837730 ; log likelihood = 0.000000
MISS 141
HIT 149 w/ (lambda (rotate_cw (rotate_cw (x_mirror (input $0))))) ; log prior = -7.783641 ; log likelihood = 0.000000
MISS 151
HIT 154 w/ (lambda (x_mirror (input $0))) ; log prior = -3.891820 ; log likelihood = 0.000000
MISS 163
HIT 171 w/ (lambda (combine_grids_vertically (input $0) (x_mirror (input $0)))) ; log prior = -7.783641 ; log likelihood = 0.000000
HIT 176 w/ (lambda (object (rotate_cw (rotate_cw (x_mirror (input $0)))))) ; log prior = -9.729551 ; log likelihood = 0.000000
HIT 178 w/ (lambda (rotate_cw (x_mirror (input $0)))) ; log prior = -5.837730 ; log likelihood = 0.000000
MISS 179
HIT 209 w/ (lambda (combine_grids_vertically (input $0) (x_mirror (input $0)))) ; log prior = -7.783641 ; log likelihood = 0.000000
MISS 210
HIT 240 w/ (lambda (rotate_cw (x_mirror (input $0)))) ; log prior = -5.837730 ; log likelihood = 0.000000
MISS 241
MISS 243
MISS 248
MISS 310
MISS 345
MISS 359
MISS 360
HIT 379 w/ (lambda (rotate_cw (rotate_cw (rotate_cw (input $0))))) ; log prior = -7.783641 ; log likelihood = 0.000000
MISS 371
HIT 384 w/ (lambda (overlay (input $0) (x_mirror (input $0)))) ; log prior = -7.783641 ; log likelihood = 0.000000
Hits 15/36 tasks
Average description length of a program solving a task: 7.005277 nats
Generative model average:  1 sec.       median: 1       max: 2  standard deviation 0
Currently using this much memory: 148295680
Frontiers discovered top down: 15
Total frontiers: 15
Currently using this much memory: 148295680
Showing the top 5 programs in each frontier being sent to the compressor:
30
-0.23   (lambda (object (input $0)))
-2.17   (lambda (object (object (input $0))))
-4.12   (lambda (object (object (object (input $0)))))
-4.12   (lambda (object (overlay (input $0) (input $0))))
-4.12   (lambda (object (x_mirror (x_mirror (input $0)))))

38
-0.36   (lambda (left_half (rotate_cw (left_half (object (input $0))))))
-2.30   (lambda (left_half (rotate_cw (left_half (object (object (input $0)))))))
-2.30   (lambda (left_half (rotate_cw (left_half (object (rotate_cw (input $0)))))))
-2.30   (lambda (left_half (rotate_cw (left_half (rotate_cw (object (input $0)))))))

86
-0.44   (lambda (rotate_cw (rotate_cw (input $0))))
-2.38   (lambda (object (rotate_cw (rotate_cw (input $0)))))
-2.38   (lambda (rotate_cw (object (rotate_cw (input $0)))))
-2.38   (lambda (rotate_cw (rotate_cw (object (input $0)))))
-4.33   (lambda (overlay (rotate_cw (rotate_cw (input $0))) (input $0)))

112
-1.06   (lambda (overlay (input $0) (x_mirror (input $0))))
-1.06   (lambda (overlay (x_mirror (input $0)) (input $0)))
-3.01   (lambda (object (overlay (input $0) (x_mirror (input $0)))))
-3.01   (lambda (object (overlay (x_mirror (input $0)) (input $0))))
-3.01   (lambda (overlay (input $0) (rotate_cw (rotate_cw (input $0)))))

115
-0.59   (lambda (combine_grids_vertically (x_mirror (input $0)) (input $0)))
-2.53   (lambda (combine_grids_vertically (object (x_mirror (input $0))) (input $0)))
-2.53   (lambda (combine_grids_vertically (x_mirror (input $0)) (object (input $0))))
-2.53   (lambda (combine_grids_vertically (x_mirror (object (input $0))) (input $0)))
-2.53   (lambda (object (combine_grids_vertically (x_mirror (input $0)) (input $0))))

139
-0.11   (lambda (rotate_cw (rotate_cw (input $0))))
-4.00   (lambda (rotate_cw (rotate_cw (overlay (input $0) (input $0)))))
-4.00   (lambda (rotate_cw (rotate_cw (x_mirror (x_mirror (input $0))))))
-4.00   (lambda (rotate_cw (x_mirror (x_mirror (rotate_cw (input $0))))))
-4.00   (lambda (x_mirror (rotate_cw (rotate_cw (x_mirror (input $0))))))

149
-1.15   (lambda (rotate_cw (rotate_cw (x_mirror (input $0)))))
-1.15   (lambda (x_mirror (rotate_cw (rotate_cw (input $0)))))
-3.09   (lambda (object (rotate_cw (rotate_cw (x_mirror (input $0))))))
-3.09   (lambda (object (x_mirror (rotate_cw (rotate_cw (input $0))))))
-3.09   (lambda (rotate_cw (object (rotate_cw (x_mirror (input $0))))))

154
-0.36   (lambda (x_mirror (input $0)))
-2.30   (lambda (object (x_mirror (input $0))))
-2.30   (lambda (x_mirror (object (input $0))))
-4.25   (lambda (object (object (x_mirror (input $0)))))
-4.25   (lambda (object (x_mirror (object (input $0)))))

171
-0.59   (lambda (combine_grids_vertically (input $0) (x_mirror (input $0))))
-2.53   (lambda (combine_grids_vertically (input $0) (object (x_mirror (input $0)))))
-2.53   (lambda (combine_grids_vertically (input $0) (x_mirror (object (input $0)))))
-2.53   (lambda (combine_grids_vertically (object (input $0)) (x_mirror (input $0))))
-2.53   (lambda (object (combine_grids_vertically (input $0) (x_mirror (input $0)))))

176
-2.11   (lambda (object (rotate_cw (rotate_cw (x_mirror (input $0))))))
-2.11   (lambda (object (x_mirror (rotate_cw (rotate_cw (input $0))))))
-2.11   (lambda (rotate_cw (object (rotate_cw (x_mirror (input $0))))))
-2.11   (lambda (rotate_cw (rotate_cw (object (x_mirror (input $0))))))
-2.11   (lambda (rotate_cw (rotate_cw (x_mirror (object (input $0))))))

178
-0.44   (lambda (rotate_cw (x_mirror (input $0))))
-2.38   (lambda (object (rotate_cw (x_mirror (input $0)))))
-2.38   (lambda (rotate_cw (object (x_mirror (input $0)))))
-2.38   (lambda (rotate_cw (x_mirror (object (input $0)))))
-4.33   (lambda (overlay (rotate_cw (x_mirror (input $0))) (input $0)))

209
-0.15   (lambda (combine_grids_vertically (input $0) (x_mirror (input $0))))
-2.10   (lambda (x_mirror (combine_grids_vertically (input $0) (x_mirror (input $0)))))
-4.04   (lambda (x_mirror (x_mirror (combine_grids_vertically (input $0) (x_mirror (input $0))))))

240
-0.44   (lambda (rotate_cw (x_mirror (input $0))))
-2.38   (lambda (object (rotate_cw (x_mirror (input $0)))))
-2.38   (lambda (rotate_cw (object (x_mirror (input $0)))))
-2.38   (lambda (rotate_cw (x_mirror (object (input $0)))))
-4.33   (lambda (rotate_cw (object (object (x_mirror (input $0))))))

379
-0.77   (lambda (rotate_cw (rotate_cw (rotate_cw (input $0)))))
-0.77   (lambda (x_mirror (rotate_cw (x_mirror (input $0)))))
-4.66   (lambda (rotate_cw (rotate_cw (rotate_cw (overlay (input $0) (input $0))))))
-4.66   (lambda (rotate_cw (rotate_cw (rotate_cw (x_mirror (x_mirror (input $0)))))))
-4.66   (lambda (rotate_cw (rotate_cw (x_mirror (x_mirror (rotate_cw (input $0)))))))

384
-0.98   (lambda (overlay (input $0) (x_mirror (input $0))))
-0.98   (lambda (overlay (x_mirror (input $0)) (input $0)))
-2.92   (lambda (object (overlay (input $0) (x_mirror (input $0)))))
-2.92   (lambda (object (overlay (x_mirror (input $0)) (input $0))))
-2.92   (lambda (x_mirror (overlay (input $0) (x_mirror (input $0)))))

Compression message saved to: compressionMessages/2022-04-28T23:22:20.223555
/home/sundong/arc/ARC_bidir_synth_supplementary_material/ec/dreamcoder/../compression: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by /home/sundong/arc/ARC_bidir_synth_supplementary_material/ec/dreamcoder/../compression)
/home/sundong/arc/ARC_bidir_synth_supplementary_material/ec/dreamcoder/../compression: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.28' not found (required by /home/sundong/arc/ARC_bidir_synth_supplementary_material/ec/dreamcoder/../compression)
Induced a grammar in 0.1 seconds
Traceback (most recent call last):
  File "bin/official_arc_symmetry.py", line 72, in <module>
    for i, result in enumerate(generator):
  File "/home/sundong/arc/ARC_bidir_synth_supplementary_material/ec/dreamcoder/dreamcoder.py", line 472, in ecIterator
    iteration=j)
  File "/home/sundong/arc/ARC_bidir_synth_supplementary_material/ec/dreamcoder/dreamcoder.py", line 693, in consolidate
    backend=compressor, CPUs=CPUs, iteration=iteration)
  File "/home/sundong/arc/ARC_bidir_synth_supplementary_material/ec/dreamcoder/compression.py", line 55, in induceGrammar
    g, newFrontiers = ocamlInduce(*args, factored=(backend == "vs_factored"), **kwargs)
  File "/home/sundong/arc/ARC_bidir_synth_supplementary_material/ec/dreamcoder/compression.py", line 159, in ocamlInduce
    response = json.loads(response.decode("utf-8"))
  File "/usr/local/conda/lib/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/usr/local/conda/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/conda/lib/python3.7/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
$ singularity --version
singularity-ce version 3.9.6

$ rpm -q glibc
glibc-2.17-325.el7_9.x86_64
glibc-2.17-325.el7_9.i686

$ cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
simonalford42 commented 2 years ago

I don't think it's an issue with GLIBC. Did you compile the ocaml before running? See https://github.com/ellisk42/ec#building-the-ocaml-binaries.

Seondong commented 2 years ago

@simonalford42 Which version of JBuilder or dune did you use to compile ocaml? Building with dune 3.1 requires lots of errors to handle.

simonalford42 commented 2 years ago

We used whatever came with the singularity image from the DreamCoder repo... are you using Singularity?

Seondong commented 2 years ago

@simonalford42 Yes, I observed the same JSONDecodeError, while running the same code in Singularity. Following your instruction, I tried to recompile the ocaml, make clean and make. But, I am failing to rebuild them.

The error message is as follows, facing the same situation with this post.

Error: the code back-end requires the type of every nonterminal symbol to be
known. Please specify the type of every symbol via %type declarations, or
enable type inference (look up --infer in the manual).
Type inference is automatically enabled when Menhir is used via Dune,
provided the dune-project file says (using menhir 2.0) or later.
The types of the following nonterminal symbols are unknown:
...

Any help appreciated!

Singularity> make clean
cd solvers && jbuilder clean
rm -f solver
rm -f compression
rm -f helmholtz
rm -f logoDrawString
rm -f data/geom/logoDrawString
Singularity> make
rm -f data/geom/logoDrawString
cd solvers && \
  jbuilder build solver.exe && \
  jbuilder build versionDemo.exe && \
  jbuilder build helmholtz.exe && \
  jbuilder build logoDrawString.exe && \
  jbuilder build protonet-tester.exe && \
  jbuilder build compression.exe && \
  jbuilder build evolution.exe && \
  cp _build/default/compression.exe ../compression && \
  cp _build/default/versionDemo.exe ../versionDemo && \
  cp _build/default/evolution.exe ../evolution && \
  cp _build/default/solver.exe ../solver && \
  cp _build/default/helmholtz.exe ../helmholtz && \
  cp _build/default/protonet-tester.exe ../protonet-tester && \
  cp _build/default/logoDrawString.exe \
    ../logoDrawString # && \
  # ln -s ../../logoDrawString \
  #  ../data/geom/logoDrawString
      menhir geomLib/geomParser.{ml,mli} (exit 1)
(cd _build/default/geomLib && /container/.opam/4.06.1+flambda/bin/menhir geomParser.mly)
Warning: 4 states have shift/reduce conflicts.
Warning: 4 shift/reduce conflicts were arbitrarily resolved.
Error: the code back-end requires the type of every nonterminal symbol to be
known. Please specify the type of every symbol via %type declarations, or
enable type inference (look up --infer in the manual).
Type inference is automatically enabled when Menhir is used via Dune,
provided the dune-project file says (using menhir 2.0) or later.
The types of the following nonterminal symbols are unknown:
expr
integrate
optional_comma
optional_integrate_accel
optional_integrate_angularAccel
optional_integrate_angularSpeed
optional_integrate_args
optional_integrate_d
optional_integrate_pen
optional_integrate_speed
optional_repeat_args
optional_turn_args
repeat
turn
value
      menhir geomLib/lambdaParser.{ml,mli} (exit 1)
(cd _build/default/geomLib && /container/.opam/4.06.1+flambda/bin/menhir lambdaParser.mly)
Error: the code back-end requires the type of every nonterminal symbol to be
known. Please specify the type of every symbol via %type declarations, or
enable type inference (look up --infer in the manual).
Type inference is automatically enabled when Menhir is used via Dune,
provided the dune-project file says (using menhir 2.0) or later.
The types of the following nonterminal symbols are unknown:
expr
someb
somev
var
      ocamlc logoLib/.logoLib.objs/logoLib__VGWrapper.{cmi,cmo,cmt}
File "logoLib/VGWrapper.ml", line 23, characters 13-15:
Warning 3: deprecated: Vg.>>
Use |> instead.
File "logoLib/VGWrapper.ml", line 26, characters 37-39:
Warning 3: deprecated: Vg.>>
Use |> instead.
File "logoLib/VGWrapper.ml", line 26, characters 12-14:
Warning 3: deprecated: Vg.>>
Use |> instead.
File "logoLib/VGWrapper.ml", line 28, characters 37-39:
Warning 3: deprecated: Vg.>>
Use |> instead.
File "logoLib/VGWrapper.ml", line 28, characters 12-14:
Warning 3: deprecated: Vg.>>
Use |> instead.
    ocamlopt logoLib/.logoLib.objs/logoLib__VGWrapper.{cmx,o}
File "logoLib/VGWrapper.ml", line 23, characters 13-15:
Warning 3: deprecated: Vg.>>
Use |> instead.
File "logoLib/VGWrapper.ml", line 26, characters 37-39:
Warning 3: deprecated: Vg.>>
Use |> instead.
File "logoLib/VGWrapper.ml", line 26, characters 12-14:
Warning 3: deprecated: Vg.>>
Use |> instead.
File "logoLib/VGWrapper.ml", line 28, characters 37-39:
Warning 3: deprecated: Vg.>>
Use |> instead.
File "logoLib/VGWrapper.ml", line 28, characters 12-14:
Warning 3: deprecated: Vg.>>
Use |> instead.
         ppx logoTest.pp.ml
File "logoTest.ml", line 67, characters 4-7:
Warning 1: this is the start of a comment.
File "logoTest.ml", line 79, characters 8-11:
Warning 1: this is the start of a comment.
File "logoTest.ml", line 80, characters 4-7:
Warning 1: this is the start of a comment.
File "logoTest.ml", line 86, characters 4-7:
Warning 1: this is the start of a comment.
make: *** [Makefile:3: all] Error 1
Singularity>
Seondong commented 2 years ago

It is somehow working now