Build & test checker without Nix

[utdemir]

To play with this, obtain earthly (or you can do nix-shell -p earthly if you have Nix), and build the targets like:

earthly +build-ligo  

[github-actions[bot]]

Gas costs: No change. Entrypoint sizes: No change. Test coverage: No change.

[github-actions[bot]]

Gas costs: No change. Entrypoint sizes: No change. Test coverage: No change.

[purcell]

I tried the end-to-end target and seem to have run into an error:

             ongoing | +tezos-binaries (2 minutes ago)
     +tezos-binaries | --> RUN source $HOME/.cargo/env; opam exec -- make build-sandbox
     +tezos-binaries | [WARNING] Running as root is not recommended
     +tezos-binaries | [WARNING] Running as root is not recommended
             ongoing |
     +tezos-binaries | --> SAVE ARTIFACT tezos-* +tezos-binaries/tezos-*
             ongoing |
                +e2e | --> COPY +tezos-binaries/* /usr/bin/
                +e2e | --> COPY (E2E_TESTS_HACK=true) +build-ligo/ ./generated/michelson
                +e2e | --> RUN poetry run python ./e2e/main.py
                +e2e | Flextesa:  Clearing root: `/tmp/checker-e2e-container-200009ll7kfr0`
                +e2e | Flextesa:
                +e2e |   Genesis-block-hash not set, using:
                +e2e |   `BLdZYwNF8Rn6zrTWkuRRNyrj6bQWPkfBog2YKhWhn5z3ApmpzBf` (corresponding
                +e2e |   chain-id: `NetXKMbjQL2SBox`) (from user choice: “Default”).
                +e2e | Flextesa:  Starting up the network.
                +e2e | Flextesa:
                +e2e |   Activating protocol ProtoALphaALphaALphaALphaALphaALphaALphaALphaDdp3zK

                +e2e | Flextesa:  Checking for all levels to be ≥ 1 (nodes: N000)
                +e2e | Flextesa:  Network started, preparing scenario.
                +e2e | Flextesa: Registration-as-delegate:  Client: "C-N000", Key: "bootacc-0"
                +e2e | Flextesa: Starting daemons:  Client: "C-N000", Key: "bootacc-0"
                +e2e | Flextesa: Registration-as-delegate:  Client: "C-N000", Key: "bootacc-1"
                +e2e | Flextesa: Starting daemons:  Client: "C-N000", Key: "bootacc-1"
                +e2e | Flextesa: Registration-as-delegate:  Client: "C-N000", Key: "bootacc-2"
                +e2e | Flextesa: Starting daemons:  Client: "C-N000", Key: "bootacc-2"
                +e2e | Flextesa: Registration-as-delegate:  Client: "C-N000", Key: "bootacc-3"
                +e2e | Flextesa: Starting daemons:  Client: "C-N000", Key: "bootacc-3"
                +e2e | Flextesa:  Checking for all levels to be ≥ 200000000 (nodes: N000)
                +e2e | Flextesa:
                +e2e |   Waiting for N000 (2) to reach level ≥ 200000000: attempt 1/200000000,
                +e2e |   sleeping 3.00 seconds

                +e2e | /usr/lib/python3.9/tempfile.py:956: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmp8f6oivbo-checker-ctez'>
                +e2e |   _warnings.warn(warn_message, ResourceWarning)
                +e2e | EFlextesa:  Clearing root: `/tmp/checker-e2e-container-20000u_5_xg84`
                +e2e | Flextesa:
                +e2e |   Genesis-block-hash not set, using:
                +e2e |   `BLdZYwNF8Rn6zrTWkuRRNyrj6bQWPkfBog2YKhWhn5z3ApmpzBf` (corresponding
                +e2e |   chain-id: `NetXKMbjQL2SBox`) (from user choice: “Default”).
                +e2e | Flextesa:  Starting up the network.
                +e2e | Flextesa:
                +e2e |   Activating protocol ProtoALphaALphaALphaALphaALphaALphaALphaALphaDdp3zK

                +e2e | Flextesa:
                +e2e |   Error:
                +e2e |     Command exited with 1
                +e2e |     Stdout: NONE
                +e2e |     Stderr:
                +e2e |
                +e2e | ````````````````````````````````````````````````````````````````````````````````
                +e2e | Warning:  the --addr --port --tls options are now deprecated; use --endpoint instead
                +e2e | Warning:
                +e2e |
                +e2e |                  This is NOT the Tezos Mainnet.
                +e2e |
                +e2e |            Do NOT use your fundraiser keys on this network.

                +e2e | Error:
                +e2e |   Fitness too low
                +e2e | ````````````````````````````````````````````````````````````````````````````````

[utdemir]

Yes, that actually is great news, because that's where I get stuck too. Means that all the stuff builds fine on Darwin.

I think the issue is that we're shutting down a tezos-sandbox executable and spawning another one right away; and there're some leftover processes from the former that clashes. I can take a look at it later this week, it should be an exercise of figuring out how to gracefully terminate the tezos-sandbox executable rather than calling .kill() on the process handle on the Python side.

[dorranh]

Not getting a ton of cache hits (build time is 3x as long as the old nix build) but things are more or less running on CI now. Once we improve the caching a bit I think this should be good to go.

[gkaracha]

Also we have to re-add a reference to earthly +spec in the README.

By the way, I can confirm that the following are working locally as expected: +build-ocaml, +builld-ligo, +test, +format, +test-coverage, +docs, +test-e2e, +test-mutations.

However, poetry install still gives me that error:

  • Installing secp256k1 (0.13.2): Failed

  EnvCommandError

  Command ['/root/.cache/pypoetry/virtualenvs/checker-SS4sQMzU-py3.9/bin/pip', 'install', '--no-deps', 'file:///root/.cache/pypoetry/artifacts/56/6a/3f/ae018aa301e6b24e329464ee2a917b748e95830ba5a3c8f3a40255cfd6/secp256k1-0.13.2.tar.gz'] errored with the following return code 1, and output: 
  Processing /root/.cache/pypoetry/artifacts/56/6a/3f/ae018aa301e6b24e329464ee2a917b748e95830ba5a3c8f3a40255cfd6/secp256k1-0.13.2.tar.gz
  Building wheels for collected packages: secp256k1
    Building wheel for secp256k1 (setup.py): started
    Building wheel for secp256k1 (setup.py): finished with status 'error'
    ERROR: Command errored out with exit status 1:
     command: /root/.cache/pypoetry/virtualenvs/checker-SS4sQMzU-py3.9/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-9fpxq40i/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-9fpxq40i/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-y1n1oxlt
         cwd: /tmp/pip-req-build-9fpxq40i/
    Complete output (50 lines):
    0.29.2
    Using bundled libsecp256k1
    running bdist_wheel
    The [wheel] section is deprecated. Use [bdist_wheel] instead.
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.9
    creating build/lib.linux-x86_64-3.9/secp256k1
    copying secp256k1/__main__.py -> build/lib.linux-x86_64-3.9/secp256k1
    copying secp256k1/__init__.py -> build/lib.linux-x86_64-3.9/secp256k1
    running build_clib
    /usr/bin/autoconf: This script requires a shell more modern than all
    /usr/bin/autoconf: the shells that I found on your system.
    /usr/bin/autoconf: Please tell bug-autoconf@gnu.org about your system,
    /usr/bin/autoconf: including any error possibly output before this
    /usr/bin/autoconf: message. Then install a modern shell, or manually run
    /usr/bin/autoconf: the script under such a shell if you do have one.
    autoreconf: /usr/bin/autoconf failed with exit status: 1
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-req-build-9fpxq40i/setup.py", line 256, in <module>
        setup(
      File "/root/.cache/pypoetry/virtualenvs/checker-SS4sQMzU-py3.9/lib/python3.9/site-packages/setuptools/__init__.py", line 153, in setup
        return distutils.core.setup(**attrs)
      File "/usr/lib/python3.9/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/usr/lib/python3.9/distutils/dist.py", line 966, in run_commands
        self.run_command(cmd)
      File "/usr/lib/python3.9/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/tmp/pip-req-build-9fpxq40i/setup.py", line 103, in run
        _bdist_wheel.run(self)
      File "/root/.cache/pypoetry/virtualenvs/checker-SS4sQMzU-py3.9/lib/python3.9/site-packages/wheel/bdist_wheel.py", line 299, in run
        self.run_command('build')
      File "/usr/lib/python3.9/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/usr/lib/python3.9/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/usr/lib/python3.9/distutils/command/build.py", line 135, in run
        self.run_command(cmd_name)
      File "/usr/lib/python3.9/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/usr/lib/python3.9/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/tmp/pip-req-build-9fpxq40i/setup.py", line 163, in run
        subprocess.check_call(
      File "/usr/lib/python3.9/subprocess.py", line 373, in check_call
        raise CalledProcessError(retcode, cmd)
    subprocess.CalledProcessError: Command '['/tmp/pip-req-build-9fpxq40i/libsecp256k1/autogen.sh']' returned non-zero exit status 1.
    ----------------------------------------
    ERROR: Failed building wheel for secp256k1
    Running setup.py clean for secp256k1
  Failed to build secp256k1
  Installing collected packages: secp256k1
      Running setup.py install for secp256k1: started
      Running setup.py install for secp256k1: finished with status 'error'
      ERROR: Command errored out with exit status 1:
       command: /root/.cache/pypoetry/virtualenvs/checker-SS4sQMzU-py3.9/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-9fpxq40i/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-9fpxq40i/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-f7p4ts70/install-record.txt --single-version-externally-managed --compile --install-headers /root/.cache/pypoetry/virtualenvs/checker-SS4sQMzU-py3.9/include/site/python3.9/secp256k1
           cwd: /tmp/pip-req-build-9fpxq40i/
      Complete output (49 lines):
      0.29.2
      Using bundled libsecp256k1
      running install
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-3.9
      creating build/lib.linux-x86_64-3.9/secp256k1
      copying secp256k1/__main__.py -> build/lib.linux-x86_64-3.9/secp256k1
      copying secp256k1/__init__.py -> build/lib.linux-x86_64-3.9/secp256k1
      running build_clib
      /usr/bin/autoconf: This script requires a shell more modern than all
      /usr/bin/autoconf: the shells that I found on your system.
      /usr/bin/autoconf: Please tell bug-autoconf@gnu.org about your system,
      /usr/bin/autoconf: including any error possibly output before this
      /usr/bin/autoconf: message. Then install a modern shell, or manually run
      /usr/bin/autoconf: the script under such a shell if you do have one.
      autoreconf: /usr/bin/autoconf failed with exit status: 1
      Traceback (most recent call last):
        File "<string>", line 1, in <module>
        File "/tmp/pip-req-build-9fpxq40i/setup.py", line 256, in <module>
          setup(
        File "/root/.cache/pypoetry/virtualenvs/checker-SS4sQMzU-py3.9/lib/python3.9/site-packages/setuptools/__init__.py", line 153, in setup
          return distutils.core.setup(**attrs)
        File "/usr/lib/python3.9/distutils/core.py", line 148, in setup
          dist.run_commands()
        File "/usr/lib/python3.9/distutils/dist.py", line 966, in run_commands
          self.run_command(cmd)
        File "/usr/lib/python3.9/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/root/.cache/pypoetry/virtualenvs/checker-SS4sQMzU-py3.9/lib/python3.9/site-packages/setuptools/command/install.py", line 61, in run
          return orig.install.run(self)
        File "/usr/lib/python3.9/distutils/command/install.py", line 590, in run
          self.run_command('build')
        File "/usr/lib/python3.9/distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/usr/lib/python3.9/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/usr/lib/python3.9/distutils/command/build.py", line 135, in run
          self.run_command(cmd_name)
        File "/usr/lib/python3.9/distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/usr/lib/python3.9/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/tmp/pip-req-build-9fpxq40i/setup.py", line 163, in run
          subprocess.check_call(
        File "/usr/lib/python3.9/subprocess.py", line 373, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['/tmp/pip-req-build-9fpxq40i/libsecp256k1/autogen.sh']' returned non-zero exit status 1.
      ----------------------------------------
  ERROR: Command errored out with exit status 1: /root/.cache/pypoetry/virtualenvs/checker-SS4sQMzU-py3.9/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-9fpxq40i/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-9fpxq40i/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-f7p4ts70/install-record.txt --single-version-externally-managed --compile --install-headers /root/.cache/pypoetry/virtualenvs/checker-SS4sQMzU-py3.9/include/site/python3.9/secp256k1 Check the logs for full command output.
  WARNING: You are using pip version 21.2.3; however, version 21.2.4 is available.
  You should consider upgrading via the '/root/.cache/pypoetry/virtualenvs/checker-SS4sQMzU-py3.9/bin/python -m pip install --upgrade pip' command.

  at /usr/local/lib/python3.9/dist-packages/poetry/utils/env.py:1101 in _run
      1097│                 output = subprocess.check_output(
      1098│                     cmd, stderr=subprocess.STDOUT, **kwargs
      1099│                 )
      1100│         except CalledProcessError as e:
    → 1101│             raise EnvCommandError(e, input=input_)
      1102│ 
      1103│         return decode(output)
      1104│ 
      1105│     def execute(self, bin, *args, **kwargs):

[dorranh]

Nice, glad to hear most of the targets are working for you! There's currently a line about the spec under the #Documentation section (similar to what's on the current master README), but I can also add another reference to it in the dev section.

Let's take another stab at debugging the dev environment on your machine. I sadly still have not been able to reproduce the issue :slightly_frowning_face:

[gkaracha]

There's currently a line about the spec under the #Documentation section (similar to what's on the current master README), but I can also add another reference to it in the dev section.

Ah, I missed that! Ignore my comment then, that reference should be sufficient :ok_hand:

Let's take another stab at debugging the dev environment on your machine. I sadly still have not been able to reproduce the issue

:+1: