ERROR: failed to solve: process "/bin/sh -c poetry install" did not complete successfully: exit code: 1

ibmgeniuz commented 4 months ago

I tried to set up gl-testing on my machine following this documentation https://blockstream.github.io/greenlight/tutorials/testing/. I always got stuck with the error below. I did not make any modifications. Any idea why this may be the case?

Steps to reproduce

Clone the github repository
Run make docker-image

Computer Specs

M2 MacBook Air
Docker 4.29.0

192.8   error: failed to run custom build command for `cln-grpc v0.1.8`
192.8
192.8   Caused by:
192.8     process didn't exit successfully: `/repo/target/release/build/cln-grpc-ef2f5f5edc573bda/build-script-build` (exit status: 101)
192.8     --- stdout
192.8     cargo:rerun-if-changed=proto/node.proto
192.8     cargo:rerun-if-changed=proto
192.8
192.8     --- stderr
192.8     thread 'main' panicked at /opt/cargo/registry/src/index.crates.io-6f17d22bba15001f/cln-grpc-0.1.8/build.rs:7:10:
192.8     called `Result::unwrap()` on an `Err` value: Custom { kind: Other, error: "protoc failed: rosetta error: failed to open elf at /lib64/ld-linux-x86-64.so.2\n " }

192.8     note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
192.8   warning: build failed, waiting for other jobs to finish...
192.8   💥 maturin failed
192.8     Caused by: Failed to build a native library through cargo
192.8     Caused by: Cargo build finished with "exit status: 101": `"cargo" "rustc" "--message-format" "json-render-diagnostics" "--manifest-path" "/repo/libs/gl-client-py/Cargo.toml" "--release" "--lib"`
192.8   Error: command ['maturin', 'pep517', 'build-wheel', '-i', '/tmp/tmpompj7ofu/.venv/bin/python', '--compatibility', 'off', '--editable'] returned non-zero exit status 1
192.8
192.8
192.8   at /tmp/poetry/venv/lib/python3.8/site-packages/poetry/installation/chef.py:164 in _prepare
192.8       160│
192.8       161│                 error = ChefBuildError("\n\n".join(message_parts))
192.8       162│
192.8       163│             if error is not None:
192.8     → 164│                 raise error from None
192.8       165│
192.8       166│             return path
192.8       167│
192.8       168│     def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:
192.8
192.8 Note: This error originates from the build backend, and is likely not a problem with poetry but with gl-client (0.1.11 /repo/libs/gl-client-py) not supporting PEP 517 builds. You can verify this by running 'pip wheel --no-cache-dir --use-pep517 --editable "/repo/libs/gl-client-py"'.
192.8
------
Dockerfile:274
--------------------
 272 |
 273 |     # Add the remaining repositories to the python-path
 274 | >>> RUN poetry install
 275 |     RUN chmod -R a+rw /tmp/venv
 276 |
--------------------

ERROR: failed to solve: process "/bin/sh -c poetry install" did not complete successfully: exit code: 1

View build details: docker-desktop://dashboard/build/desktop-linux/desktop-linux/ql3hsngo5lnht6va2vfdbnr8l
make: *** [docker-image] Error 1

ErikDeSmedt commented 4 months ago

I suspect the issue is that one of our dependencies does not support the M2 processor architecture. We download a binary for protoc that can only be used on x86_64.

As a work-around you could try and use a version that is compiled for Mac M2. You can do this by editin ./docker/gl-testing/Dockerfile in line 163. You replace the url to a binary that works on your machine.

I don't have a Mac with M2-architecture so I can't test locally. Let me know if you need more details or help

cdecker commented 4 months ago

Hm, that's very strange though, it talks about rosetta which is the CPU virtualization framework used to emulate x86 on arm64 and exists solely for Mac OS. This is inside of Docker however, which is an emulated intel linux machine iirc, so how can it end up trying to execute an intel binary inside the docker image, with the emulator on the host?

I'm confused tbh. But maybe it is actually emulating an arm64 linux machine? That would explain the mismatch, but not how the host emulator is involved.

ibmgeniuz commented 4 months ago

@ErikDeSmedt Are you talking about the releases on this page https://github.com/protocolbuffers/protobuf/releases ? What is in the Dockerfile seem to be pretty outdated compared to what I see on that page. https://github.com/protocolbuffers/protobuf/releases/download/v3.19.3/protoc-3.19.3-linux-x86_64.zip but I see v26.1 . Am I on the right page?

Anyway I changed the linux-x86_64 part to linux-aarch_64 . It's been running for more than 1200 secs at 54/56. So far it has not given the error (fingers crossed) but I don't see progress yet. I will let you know how it goes.

@cdecker I was also thinking the same. I don't have rosetta on my Mac. However, if the whole idea of using docker is to ensure that we don't have to install dependencies on our computer, then it's surprising to me that I would need to install rosetta. I've held off installing rosetta on my computer for a very long time so it will be a real bummer if I would need to install it before this can work.

ibmgeniuz commented 4 months ago

@ErikDeSmedt @cdecker the command make docker-image completed successfully after I replaced https://github.com/protocolbuffers/protobuf/releases/download/v3.19.3/protoc-3.19.3-linux-x86_64.zip with https://github.com/protocolbuffers/protobuf/releases/download/v3.19.3/protoc-3.19.3-linux-aarch_64.zip . I went ahead to run

make docker-shell
exited the shell.
created my-first-test.py at the root of the repository gl-testing-tutorial with contents same as in the tutorial
executed make build-self after make docker-shell
Then I ran pytest -v -s my-first-test.py but got the error below.

Any idea why this is the case? Do I need to create a separate issue?

―――――――――――――――――――――――――――――― ERROR at setup of test_invoice_payment ――――――――――――――――――――――――――――――

directory = PosixPath('/tmp/ltests-vbm295qu/test_invoice_payment_1/gl-testing')
teardown_checks = <pyln.testing.fixtures.TeardownErrors object at 0xffff836377c0>

    @pytest.fixture
    def bitcoind(directory, teardown_checks):
        chaind = network_daemons[env('TEST_NETWORK', 'regtest')]
        bitcoind = chaind(bitcoin_dir=directory)

        try:
>           bitcoind.start()

/tmp/venv/lib/python3.8/site-packages/pyln/testing/fixtures.py:132:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/tmp/venv/lib/python3.8/site-packages/pyln/testing/utils.py:437: in start
    self.wait_for_log("Done loading", timeout=TIMEOUT)
/tmp/venv/lib/python3.8/site-packages/pyln/testing/utils.py:347: in wait_for_log
    return self.wait_for_logs([regex], timeout)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <pyln.testing.utils.BitcoinD object at 0xffff83637670>, regexs = ['Done loading']
timeout = 60

    def wait_for_logs(self, regexs, timeout=TIMEOUT):
        """Look for `regexs` in the logs.

        The logs contain tailed stdout of the process. We look for each regex
        in `regexs`, starting from `logsearch_start` which normally is the
        position of the last found entry of a previous wait-for logs call.
        The ordering inside `regexs` doesn't matter.

        We fail if the timeout is exceeded or if the underlying process
        exits before all the `regexs` were found.

        If timeout is None, no time-out is applied.
        """
        logging.debug("Waiting for {} in the logs".format(regexs))
        exs = [re.compile(r) for r in regexs]
        start_time = time.time()
        while True:
            if self.logsearch_start >= len(self.logs):
                if not self.logs_catchup():
                    time.sleep(0.25)

                if timeout is not None and time.time() > start_time + timeout:
                    print("Time-out: can't find {} in logs".format(exs))
                    for r in exs:
                        if self.is_in_log(r):
                            print("({} was previously in logs!)".format(r))
>                   raise TimeoutError('Unable to find "{}" in logs.'.format(exs))
E                   TimeoutError: Unable to find "[re.compile('Done loading')]" in logs.

/tmp/venv/lib/python3.8/site-packages/pyln/testing/utils.py:328: TimeoutError

During handling of the above exception, another exception occurred:

>       lambda: ihook(item=item, **kwds), when=when, reraise=reraise
    )

/tmp/venv/lib/python3.8/site-packages/flaky/flaky_pytest_plugin.py:146:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/tmp/venv/lib/python3.8/site-packages/pyln/testing/fixtures.py:134: in bitcoind
    bitcoind.stop()
/tmp/venv/lib/python3.8/site-packages/pyln/testing/utils.py:448: in stop
    self.rpc.stop()
/tmp/venv/lib/python3.8/site-packages/pyln/testing/utils.py:374: in f
    res = proxy._call(name, *args)
/tmp/venv/lib/python3.8/site-packages/bitcoin/rpc.py:244: in _call
    self.__conn.request('POST', self.__url.path, postdata, headers)
/usr/lib/python3.8/http/client.py:1256: in request
    self._send_request(method, url, body, headers, encode_chunked)
/usr/lib/python3.8/http/client.py:1302: in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
/usr/lib/python3.8/http/client.py:1251: in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
/usr/lib/python3.8/http/client.py:1011: in _send_output
    self.send(msg)
/usr/lib/python3.8/http/client.py:951: in send
    self.connect()
/usr/lib/python3.8/http/client.py:922: in connect
    self.sock = self._create_connection(
/usr/lib/python3.8/socket.py:808: in create_connection
    raise err
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

address = ('localhost', 45091), timeout = 30, source_address = None

    def create_connection(address, timeout=_GLOBAL_DEFAULT_TIMEOUT,
                          source_address=None):
        """Connect to *address* and return the socket object.

        Convenience function.  Connect to *address* (a 2-tuple ``(host,
        port)``) and return the socket object.  Passing the optional
        *timeout* parameter will set the timeout on the socket instance
        before attempting to connect.  If no *timeout* is supplied, the
        global default timeout setting returned by :func:`getdefaulttimeout`
        is used.  If *source_address* is set it must be a tuple of (host, port)
        for the socket to bind as a source address before making the connection.
        A host of '' or port 0 tells the OS to use the default.
        """

        host, port = address
        err = None
        for res in getaddrinfo(host, port, 0, SOCK_STREAM):
            af, socktype, proto, canonname, sa = res
            sock = None
            try:
                sock = socket(af, socktype, proto)
                if timeout is not _GLOBAL_DEFAULT_TIMEOUT:
                    sock.settimeout(timeout)
                if source_address:
                    sock.bind(source_address)
>               sock.connect(sa)
E               ConnectionRefusedError: [Errno 111] Connection refused

/usr/lib/python3.8/socket.py:796: ConnectionRefusedError
---------------------------------------- Captured log setup ----------------------------------------
DEBUG    root:utils.py:225 Starting 'bitcoind -datadir=/tmp/ltests-vbm295qu/test_invoice_payment_1/gl-testing -printtoconsole -server -logtimestamps -nolisten -txindex -nowallet -addresstype=bech32 -debug=mempool -debug=mempoolrej -debug=rpc -debug=validation'
DEBUG    root:utils.py:315 Waiting for ['Done loading'] in the logs
DEBUG    root:utils.py:285 Did not find 're.compile('Done loading')' in logs
DEBUG    root:utils.py:370 Calling stop with arguments ()
                                                                                     100% ██████████
========================================= warnings summary =========================================
../tmp/venv/lib/python3.8/site-packages/cheroot/__init__.py:7
  /tmp/venv/lib/python3.8/site-packages/cheroot/__init__.py:7: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
    import pkg_resources

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
===================================== short test summary info ======================================
FAILED my-first-test.py::test_invoice_payment - ConnectionRefusedError: [Errno 111] Connection refused

Results (61.01s (0:01:01)):
       1 error

ErikDeSmedt commented 4 months ago

I have no experience building on Mac.

However, I think you might be in for a rough ride. Currently we only support x86_64.

The problem appears to be that bitcoind doesnt't start. You could try to download a bitcoind that is suitable for your architecture and try again.

The bigger problem is that you will also need a wide range of Core Lightning versions. For those is no M2 version available and building them yourself would be very tricky.

On Fri, Apr 19, 2024, 5:39 AM Issah Bin Mohammed @.***> wrote:

@ErikDeSmedt https://github.com/ErikDeSmedt @cdecker https://github.com/cdecker the command make docker-image completed successfully after I replaced https://github.com/protocolbuffers/protobuf/releases/download/v3.19.3/protoc-3.19.3-linux-x86_64.zip with https://github.com/protocolbuffers/protobuf/releases/download/v3.19.3/protoc-3.19.3-linux-aarch_64.zip . I went ahead to run

make docker-shell

exited the shell.

created my-first-test.py at the root of the repository gl-testing-tutorial with contents same as in the tutorial https://blockstream.github.io/greenlight/tutorials/testing/

executed make build-self after make docker-shell

Then I ran pytest -v -s my-first-test.py but got the error below.

Any idea why this is the case? Do I need to create a separate issue?

―――――――――――――――――――――――――――――― ERROR at setup of test_invoice_payment ――――――――――――――――――――――――――――――

directory = PosixPath('/tmp/ltests-vbm295qu/test_invoice_payment_1/gl-testing') teardown_checks = <pyln.testing.fixtures.TeardownErrors object at 0xffff836377c0>
@pytest.fixture
def bitcoind(directory, teardown_checks):
    chaind = network_daemons[env('TEST_NETWORK', 'regtest')]
    bitcoind = chaind(bitcoin_dir=directory)

    try:>           bitcoind.start()
/tmp/venv/lib/python3.8/site-packages/pyln/testing/fixtures.py:132:

/tmp/venv/lib/python3.8/site-packages/pyln/testing/utils.py:437: in start self.wait_for_log("Done loading", timeout=TIMEOUT) /tmp/venv/lib/python3.8/site-packages/pyln/testing/utils.py:347: in wait_for_log return self.wait_for_logs([regex], timeout)

self = <pyln.testing.utils.BitcoinD object at 0xffff83637670>, regexs = ['Done loading'] timeout = 60
def wait_for_logs(self, regexs, timeout=TIMEOUT):
    """Look for `regexs` in the logs.        The logs contain tailed stdout of the process. We look for each regex        in `regexs`, starting from `logsearch_start` which normally is the        position of the last found entry of a previous wait-for logs call.        The ordering inside `regexs` doesn't matter.        We fail if the timeout is exceeded or if the underlying process        exits before all the `regexs` were found.        If timeout is None, no time-out is applied.        """
    logging.debug("Waiting for {} in the logs".format(regexs))
    exs = [re.compile(r) for r in regexs]
    start_time = time.time()
    while True:
        if self.logsearch_start >= len(self.logs):
            if not self.logs_catchup():
                time.sleep(0.25)

            if timeout is not None and time.time() > start_time + timeout:
                print("Time-out: can't find {} in logs".format(exs))
                for r in exs:
                    if self.is_in_log(r):
                        print("({} was previously in logs!)".format(r))>                   raise TimeoutError('Unable to find "{}" in logs.'.format(exs))
E TimeoutError: Unable to find "[re.compile('Done loading')]" in logs.

/tmp/venv/lib/python3.8/site-packages/pyln/testing/utils.py:328: TimeoutError

During handling of the above exception, another exception occurred:
  lambda: ihook(item=item, **kwds), when=when, reraise=reraise
)
/tmp/venv/lib/python3.8/site-packages/flaky/flaky_pytest_plugin.py:146:

/tmp/venv/lib/python3.8/site-packages/pyln/testing/fixtures.py:134: in bitcoind bitcoind.stop() /tmp/venv/lib/python3.8/site-packages/pyln/testing/utils.py:448: in stop self.rpc.stop() /tmp/venv/lib/python3.8/site-packages/pyln/testing/utils.py:374: in f res = proxy._call(name, *args) /tmp/venv/lib/python3.8/site-packages/bitcoin/rpc.py:244: in _call self.conn.request('POST', self.url.path, postdata, headers) /usr/lib/python3.8/http/client.py:1256: in request self._send_request(method, url, body, headers, encode_chunked) /usr/lib/python3.8/http/client.py:1302: in _send_request self.endheaders(body, encode_chunked=encode_chunked) /usr/lib/python3.8/http/client.py:1251: in endheaders self._send_output(message_body, encode_chunked=encode_chunked) /usr/lib/python3.8/http/client.py:1011: in _send_output self.send(msg) /usr/lib/python3.8/http/client.py:951: in send self.connect() /usr/lib/python3.8/http/client.py:922: in connect self.sock = self._create_connection( /usr/lib/python3.8/socket.py:808: in create_connection raise err

address = ('localhost', 45091), timeout = 30, source_address = None
def create_connection(address, timeout=_GLOBAL_DEFAULT_TIMEOUT,
                      source_address=None):
    """Connect to *address* and return the socket object.        Convenience function.  Connect to *address* (a 2-tuple ``(host,        port)``) and return the socket object.  Passing the optional        *timeout* parameter will set the timeout on the socket instance        before attempting to connect.  If no *timeout* is supplied, the        global default timeout setting returned by :func:`getdefaulttimeout`        is used.  If *source_address* is set it must be a tuple of (host, port)        for the socket to bind as a source address before making the connection.        A host of '' or port 0 tells the OS to use the default.        """

    host, port = address
    err = None
    for res in getaddrinfo(host, port, 0, SOCK_STREAM):
        af, socktype, proto, canonname, sa = res
        sock = None
        try:
            sock = socket(af, socktype, proto)
            if timeout is not _GLOBAL_DEFAULT_TIMEOUT:
                sock.settimeout(timeout)
            if source_address:
                sock.bind(source_address)>               sock.connect(sa)
E ConnectionRefusedError: [Errno 111] Connection refused

/usr/lib/python3.8/socket.py:796: ConnectionRefusedError ---------------------------------------- Captured log setup ---------------------------------------- DEBUG root:utils.py:225 Starting 'bitcoind -datadir=/tmp/ltests-vbm295qu/test_invoice_payment_1/gl-testing -printtoconsole -server -logtimestamps -nolisten -txindex -nowallet -addresstype=bech32 -debug=mempool -debug=mempoolrej -debug=rpc -debug=validation' DEBUG root:utils.py:315 Waiting for ['Done loading'] in the logs DEBUG root:utils.py:285 Did not find 're.compile('Done loading')' in logs DEBUG root:utils.py:370 Calling stop with arguments () 100% ██████████ ========================================= warnings summary ========================================= ../tmp/venv/lib/python3.8/site-packages/cheroot/init.py:7 /tmp/venv/lib/python3.8/site-packages/cheroot/init.py:7: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html import pkg_resources

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ===================================== short test summary info ====================================== FAILED my-first-test.py::test_invoice_payment - ConnectionRefusedError: [Errno 111] Connection refused

Results (61.01s (0:01:01)): 1 error

— Reply to this email directly, view it on GitHub https://github.com/Blockstream/greenlight/issues/411#issuecomment-2065690555, or unsubscribe https://github.com/notifications/unsubscribe-auth/AATJLK6LPLH2ANPXW5TCEUTY6CGXJAVCNFSM6AAAAABGLYXHO2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANRVGY4TANJVGU . You are receiving this because you were mentioned.Message ID: @.***>

ibmgeniuz commented 4 months ago

Alright. Thank you. I will see if I can hack my way around it. Whatever be the case, I will let you know. I will keep this issue opened in case we reach some solutions somehow.

I have no experience building on Mac. However, I think you might be in for a rough ride. Currently we only support x86_64. The problem appears to be that bitcoind doesnt't start. You could try to download a bitcoind that is suitable for your architecture and try again. The bigger problem is that you will also need a wide range of Core Lightning versions. For those is no M2 version available and building them yourself would be very tricky.

cdecker commented 3 months ago

Closing this one as it is more related to the dev setup being messed up between the various architectures (thanks Rosetta) and not with the software in this repo.

Blockstream / greenlight

ERROR: failed to solve: process "/bin/sh -c poetry install" did not complete successfully: exit code: 1 #411

Steps to reproduce

Computer Specs