Closed ibmgeniuz closed 3 months ago
I suspect the issue is that one of our dependencies does not support the M2
processor architecture.
We download a binary for protoc
that can only be used on x86_64
.
As a work-around you could try and use a version that is compiled for Mac M2.
You can do this by editin ./docker/gl-testing/Dockerfile
in line 163. You replace the url to a binary that works on your machine.
I don't have a Mac with M2-architecture so I can't test locally. Let me know if you need more details or help
Hm, that's very strange though, it talks about rosetta
which is the CPU virtualization framework used to emulate x86 on arm64 and exists solely for Mac OS. This is inside of Docker however, which is an emulated intel linux machine iirc, so how can it end up trying to execute an intel binary inside the docker image, with the emulator on the host?
I'm confused tbh. But maybe it is actually emulating an arm64 linux machine? That would explain the mismatch, but not how the host emulator is involved.
@ErikDeSmedt Are you talking about the releases on this page https://github.com/protocolbuffers/protobuf/releases ?
What is in the Dockerfile seem to be pretty outdated compared to what I see on that page. https://github.com/protocolbuffers/protobuf/releases/download/v3.19.3/protoc-3.19.3-linux-x86_64.zip
but I see v26.1
. Am I on the right page?
Anyway I changed the linux-x86_64
part to linux-aarch_64
. It's been running for more than 1200 secs at 54/56
. So far it has not given the error (fingers crossed) but I don't see progress yet. I will let you know how it goes.
@cdecker I was also thinking the same. I don't have rosetta on my Mac. However, if the whole idea of using docker is to ensure that we don't have to install dependencies on our computer, then it's surprising to me that I would need to install rosetta. I've held off installing rosetta on my computer for a very long time so it will be a real bummer if I would need to install it before this can work.
@ErikDeSmedt @cdecker the command make docker-image
completed successfully after I replaced https://github.com/protocolbuffers/protobuf/releases/download/v3.19.3/protoc-3.19.3-linux-x86_64.zip
with https://github.com/protocolbuffers/protobuf/releases/download/v3.19.3/protoc-3.19.3-linux-aarch_64.zip
. I went ahead to run
make docker-shell
my-first-test.py
at the root of the repository gl-testing-tutorial
with contents same as in the tutorialmake build-self
after make docker-shell
pytest -v -s my-first-test.py
but got the error below. Any idea why this is the case? Do I need to create a separate issue?
―――――――――――――――――――――――――――――― ERROR at setup of test_invoice_payment ――――――――――――――――――――――――――――――
directory = PosixPath('/tmp/ltests-vbm295qu/test_invoice_payment_1/gl-testing')
teardown_checks = <pyln.testing.fixtures.TeardownErrors object at 0xffff836377c0>
@pytest.fixture
def bitcoind(directory, teardown_checks):
chaind = network_daemons[env('TEST_NETWORK', 'regtest')]
bitcoind = chaind(bitcoin_dir=directory)
try:
> bitcoind.start()
/tmp/venv/lib/python3.8/site-packages/pyln/testing/fixtures.py:132:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/tmp/venv/lib/python3.8/site-packages/pyln/testing/utils.py:437: in start
self.wait_for_log("Done loading", timeout=TIMEOUT)
/tmp/venv/lib/python3.8/site-packages/pyln/testing/utils.py:347: in wait_for_log
return self.wait_for_logs([regex], timeout)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <pyln.testing.utils.BitcoinD object at 0xffff83637670>, regexs = ['Done loading']
timeout = 60
def wait_for_logs(self, regexs, timeout=TIMEOUT):
"""Look for `regexs` in the logs.
The logs contain tailed stdout of the process. We look for each regex
in `regexs`, starting from `logsearch_start` which normally is the
position of the last found entry of a previous wait-for logs call.
The ordering inside `regexs` doesn't matter.
We fail if the timeout is exceeded or if the underlying process
exits before all the `regexs` were found.
If timeout is None, no time-out is applied.
"""
logging.debug("Waiting for {} in the logs".format(regexs))
exs = [re.compile(r) for r in regexs]
start_time = time.time()
while True:
if self.logsearch_start >= len(self.logs):
if not self.logs_catchup():
time.sleep(0.25)
if timeout is not None and time.time() > start_time + timeout:
print("Time-out: can't find {} in logs".format(exs))
for r in exs:
if self.is_in_log(r):
print("({} was previously in logs!)".format(r))
> raise TimeoutError('Unable to find "{}" in logs.'.format(exs))
E TimeoutError: Unable to find "[re.compile('Done loading')]" in logs.
/tmp/venv/lib/python3.8/site-packages/pyln/testing/utils.py:328: TimeoutError
During handling of the above exception, another exception occurred:
> lambda: ihook(item=item, **kwds), when=when, reraise=reraise
)
/tmp/venv/lib/python3.8/site-packages/flaky/flaky_pytest_plugin.py:146:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/tmp/venv/lib/python3.8/site-packages/pyln/testing/fixtures.py:134: in bitcoind
bitcoind.stop()
/tmp/venv/lib/python3.8/site-packages/pyln/testing/utils.py:448: in stop
self.rpc.stop()
/tmp/venv/lib/python3.8/site-packages/pyln/testing/utils.py:374: in f
res = proxy._call(name, *args)
/tmp/venv/lib/python3.8/site-packages/bitcoin/rpc.py:244: in _call
self.__conn.request('POST', self.__url.path, postdata, headers)
/usr/lib/python3.8/http/client.py:1256: in request
self._send_request(method, url, body, headers, encode_chunked)
/usr/lib/python3.8/http/client.py:1302: in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
/usr/lib/python3.8/http/client.py:1251: in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
/usr/lib/python3.8/http/client.py:1011: in _send_output
self.send(msg)
/usr/lib/python3.8/http/client.py:951: in send
self.connect()
/usr/lib/python3.8/http/client.py:922: in connect
self.sock = self._create_connection(
/usr/lib/python3.8/socket.py:808: in create_connection
raise err
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
address = ('localhost', 45091), timeout = 30, source_address = None
def create_connection(address, timeout=_GLOBAL_DEFAULT_TIMEOUT,
source_address=None):
"""Connect to *address* and return the socket object.
Convenience function. Connect to *address* (a 2-tuple ``(host,
port)``) and return the socket object. Passing the optional
*timeout* parameter will set the timeout on the socket instance
before attempting to connect. If no *timeout* is supplied, the
global default timeout setting returned by :func:`getdefaulttimeout`
is used. If *source_address* is set it must be a tuple of (host, port)
for the socket to bind as a source address before making the connection.
A host of '' or port 0 tells the OS to use the default.
"""
host, port = address
err = None
for res in getaddrinfo(host, port, 0, SOCK_STREAM):
af, socktype, proto, canonname, sa = res
sock = None
try:
sock = socket(af, socktype, proto)
if timeout is not _GLOBAL_DEFAULT_TIMEOUT:
sock.settimeout(timeout)
if source_address:
sock.bind(source_address)
> sock.connect(sa)
E ConnectionRefusedError: [Errno 111] Connection refused
/usr/lib/python3.8/socket.py:796: ConnectionRefusedError
---------------------------------------- Captured log setup ----------------------------------------
DEBUG root:utils.py:225 Starting 'bitcoind -datadir=/tmp/ltests-vbm295qu/test_invoice_payment_1/gl-testing -printtoconsole -server -logtimestamps -nolisten -txindex -nowallet -addresstype=bech32 -debug=mempool -debug=mempoolrej -debug=rpc -debug=validation'
DEBUG root:utils.py:315 Waiting for ['Done loading'] in the logs
DEBUG root:utils.py:285 Did not find 're.compile('Done loading')' in logs
DEBUG root:utils.py:370 Calling stop with arguments ()
100% ██████████
========================================= warnings summary =========================================
../tmp/venv/lib/python3.8/site-packages/cheroot/__init__.py:7
/tmp/venv/lib/python3.8/site-packages/cheroot/__init__.py:7: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
import pkg_resources
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
===================================== short test summary info ======================================
FAILED my-first-test.py::test_invoice_payment - ConnectionRefusedError: [Errno 111] Connection refused
Results (61.01s (0:01:01)):
1 error
I have no experience building on Mac.
However, I think you might be in for a rough ride. Currently we only support x86_64.
The problem appears to be that bitcoind
doesnt't start. You could try to
download a bitcoind that is suitable for your architecture and try again.
The bigger problem is that you will also need a wide range of Core Lightning versions. For those is no M2 version available and building them yourself would be very tricky.
On Fri, Apr 19, 2024, 5:39 AM Issah Bin Mohammed @.***> wrote:
@ErikDeSmedt https://github.com/ErikDeSmedt @cdecker https://github.com/cdecker the command make docker-image completed successfully after I replaced https://github.com/protocolbuffers/protobuf/releases/download/v3.19.3/protoc-3.19.3-linux-x86_64.zip with https://github.com/protocolbuffers/protobuf/releases/download/v3.19.3/protoc-3.19.3-linux-aarch_64.zip . I went ahead to run
- make docker-shell
- exited the shell.
- created my-first-test.py at the root of the repository gl-testing-tutorial with contents same as in the tutorial https://blockstream.github.io/greenlight/tutorials/testing/
- executed make build-self after make docker-shell
- Then I ran pytest -v -s my-first-test.py but got the error below.
Any idea why this is the case? Do I need to create a separate issue?
―――――――――――――――――――――――――――――― ERROR at setup of test_invoice_payment ――――――――――――――――――――――――――――――
directory = PosixPath('/tmp/ltests-vbm295qu/test_invoice_payment_1/gl-testing') teardown_checks = <pyln.testing.fixtures.TeardownErrors object at 0xffff836377c0>
@pytest.fixture def bitcoind(directory, teardown_checks): chaind = network_daemons[env('TEST_NETWORK', 'regtest')] bitcoind = chaind(bitcoin_dir=directory) try:> bitcoind.start()
/tmp/venv/lib/python3.8/site-packages/pyln/testing/fixtures.py:132:
/tmp/venv/lib/python3.8/site-packages/pyln/testing/utils.py:437: in start self.wait_for_log("Done loading", timeout=TIMEOUT) /tmp/venv/lib/python3.8/site-packages/pyln/testing/utils.py:347: in wait_for_log return self.wait_for_logs([regex], timeout)
self = <pyln.testing.utils.BitcoinD object at 0xffff83637670>, regexs = ['Done loading'] timeout = 60
def wait_for_logs(self, regexs, timeout=TIMEOUT): """Look for `regexs` in the logs. The logs contain tailed stdout of the process. We look for each regex in `regexs`, starting from `logsearch_start` which normally is the position of the last found entry of a previous wait-for logs call. The ordering inside `regexs` doesn't matter. We fail if the timeout is exceeded or if the underlying process exits before all the `regexs` were found. If timeout is None, no time-out is applied. """ logging.debug("Waiting for {} in the logs".format(regexs)) exs = [re.compile(r) for r in regexs] start_time = time.time() while True: if self.logsearch_start >= len(self.logs): if not self.logs_catchup(): time.sleep(0.25) if timeout is not None and time.time() > start_time + timeout: print("Time-out: can't find {} in logs".format(exs)) for r in exs: if self.is_in_log(r): print("({} was previously in logs!)".format(r))> raise TimeoutError('Unable to find "{}" in logs.'.format(exs))
E TimeoutError: Unable to find "[re.compile('Done loading')]" in logs.
/tmp/venv/lib/python3.8/site-packages/pyln/testing/utils.py:328: TimeoutError
During handling of the above exception, another exception occurred:
lambda: ihook(item=item, **kwds), when=when, reraise=reraise
)
/tmp/venv/lib/python3.8/site-packages/flaky/flaky_pytest_plugin.py:146:
/tmp/venv/lib/python3.8/site-packages/pyln/testing/fixtures.py:134: in bitcoind bitcoind.stop() /tmp/venv/lib/python3.8/site-packages/pyln/testing/utils.py:448: in stop self.rpc.stop() /tmp/venv/lib/python3.8/site-packages/pyln/testing/utils.py:374: in f res = proxy._call(name, *args) /tmp/venv/lib/python3.8/site-packages/bitcoin/rpc.py:244: in _call self.conn.request('POST', self.url.path, postdata, headers) /usr/lib/python3.8/http/client.py:1256: in request self._send_request(method, url, body, headers, encode_chunked) /usr/lib/python3.8/http/client.py:1302: in _send_request self.endheaders(body, encode_chunked=encode_chunked) /usr/lib/python3.8/http/client.py:1251: in endheaders self._send_output(message_body, encode_chunked=encode_chunked) /usr/lib/python3.8/http/client.py:1011: in _send_output self.send(msg) /usr/lib/python3.8/http/client.py:951: in send self.connect() /usr/lib/python3.8/http/client.py:922: in connect self.sock = self._create_connection( /usr/lib/python3.8/socket.py:808: in create_connection raise err
address = ('localhost', 45091), timeout = 30, source_address = None
def create_connection(address, timeout=_GLOBAL_DEFAULT_TIMEOUT, source_address=None): """Connect to *address* and return the socket object. Convenience function. Connect to *address* (a 2-tuple ``(host, port)``) and return the socket object. Passing the optional *timeout* parameter will set the timeout on the socket instance before attempting to connect. If no *timeout* is supplied, the global default timeout setting returned by :func:`getdefaulttimeout` is used. If *source_address* is set it must be a tuple of (host, port) for the socket to bind as a source address before making the connection. A host of '' or port 0 tells the OS to use the default. """ host, port = address err = None for res in getaddrinfo(host, port, 0, SOCK_STREAM): af, socktype, proto, canonname, sa = res sock = None try: sock = socket(af, socktype, proto) if timeout is not _GLOBAL_DEFAULT_TIMEOUT: sock.settimeout(timeout) if source_address: sock.bind(source_address)> sock.connect(sa)
E ConnectionRefusedError: [Errno 111] Connection refused
/usr/lib/python3.8/socket.py:796: ConnectionRefusedError ---------------------------------------- Captured log setup ---------------------------------------- DEBUG root:utils.py:225 Starting 'bitcoind -datadir=/tmp/ltests-vbm295qu/test_invoice_payment_1/gl-testing -printtoconsole -server -logtimestamps -nolisten -txindex -nowallet -addresstype=bech32 -debug=mempool -debug=mempoolrej -debug=rpc -debug=validation' DEBUG root:utils.py:315 Waiting for ['Done loading'] in the logs DEBUG root:utils.py:285 Did not find 're.compile('Done loading')' in logs DEBUG root:utils.py:370 Calling stop with arguments () 100% ██████████ ========================================= warnings summary ========================================= ../tmp/venv/lib/python3.8/site-packages/cheroot/init.py:7 /tmp/venv/lib/python3.8/site-packages/cheroot/init.py:7: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html import pkg_resources
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ===================================== short test summary info ====================================== FAILED my-first-test.py::test_invoice_payment - ConnectionRefusedError: [Errno 111] Connection refused
Results (61.01s (0:01:01)): 1 error
— Reply to this email directly, view it on GitHub https://github.com/Blockstream/greenlight/issues/411#issuecomment-2065690555, or unsubscribe https://github.com/notifications/unsubscribe-auth/AATJLK6LPLH2ANPXW5TCEUTY6CGXJAVCNFSM6AAAAABGLYXHO2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANRVGY4TANJVGU . You are receiving this because you were mentioned.Message ID: @.***>
Alright. Thank you. I will see if I can hack my way around it. Whatever be the case, I will let you know. I will keep this issue opened in case we reach some solutions somehow.
I have no experience building on Mac. However, I think you might be in for a rough ride. Currently we only support x86_64. The problem appears to be that
bitcoind
doesnt't start. You could try to download a bitcoind that is suitable for your architecture and try again. The bigger problem is that you will also need a wide range of Core Lightning versions. For those is no M2 version available and building them yourself would be very tricky.
Closing this one as it is more related to the dev setup being messed up between the various architectures (thanks Rosetta) and not with the software in this repo.
I tried to set up
gl-testing
on my machine following this documentation https://blockstream.github.io/greenlight/tutorials/testing/. I always got stuck with the error below. I did not make any modifications. Any idea why this may be the case?Steps to reproduce
make docker-image
Computer Specs