search

localstack / localstack

💻 A fully functional local AWS cloud stack. Develop and test your cloud & Serverless apps offline
https://localstack.cloud
Other
55.6k stars 3.96k forks source link

bug: cannot run kinesis on apple M1 ARM with Docker #4728

Closed caiorcferreira closed 2 years ago

caiorcferreira commented 2 years ago

Is there an existing issue for this?

Current Behavior

The PR #4191 introduced a way for LocalStack to detect that its running on Apple M1, however this doesn't work for Docker since it will the platform as Linux x86.

I understand that provide an ARM image may not be trivial, but a simpler solution would be to have an environment variable that the user can set to force LocalStack to use Kinesis Mock JAR.

Expected Behavior

Running LocalStack's Kinesis Firehose via Docker on Apple M1 should work.

How are you starting LocalStack?

With a docker-compose file

Steps To Reproduce

How are you starting localstack (e.g., bin/localstack command, arguments, or docker-compose.yml)

docker run localstack/localstack:0.12.18 -e LOCALSTACK_SERVICES=firehose -e KINESIS_PROVIDER=kinesis-mock -e KINESIS_MOCK_PLAIN_PORT=4568 -e KINESIS_INITIALIZE_STREAMS=metaplane-audit:1 -e DEBUG=1

Environment

- OS: macOS Big Sur 11.6
- LocalStack: 0.12.18

Anything else?

No response

viswanath7 commented 2 years ago

I can confirm this bug in ARM-64 / apple silicon. I encountered the same problem. Local stack fails to start-up dynamoDb service due to kinesis.

I have the following service definiton in my docker-compose file for local stack 


localstack:
    image: localstack/localstack:0.12.18
    ports:
      - "4566:4566"
      - "4571:4571"
    environment:
      - SERVICES=sqs,dynamodb
      - DEBUG=1
      - DEFAULT_REGION=eu-west-1
      - DOCKER_HOST=unix:///var/run/docker.sock
      - HOSTNAME_EXTERNAL=localstack
    volumes:
      - ./deployment/local/aws:/docker-entrypoint-initaws.d/
      - /tmp/localstack:/tmp/localstack
      - /var/run/docker.sock:/var/run/docker.sock

and that results in the following stack trace 

 localstack_1  | 2021-10-16T18:47:58:DEBUG:bootstrap.py: Plugin loading took 8.10982894897461 sec
localstack_1  | 2021-10-16T18:47:58:INFO:bootstrap.py: Execution of "load_plugins" took 8110.66ms
localstack_1  | 2021-10-16T18:47:58:DEBUG:localstack.services.install: getting kinesis-mock for linux x86_64
localstack_1  | 2021-10-16T18:47:58:DEBUG:localstack.services.install: kinesis-mock found at /opt/code/localstack/localstack/infra/kinesis-mock/kinesis-mock-linux-amd64-static
localstack_1  | 2021-10-16T18:47:58:DEBUG:localstack.utils.run: Executing command: cd "/opt/code/localstack/localstack/infra/dynamodb" && zip -u DynamoDBLocal.jar log4j2.xml || true
localstack_1  | Starting edge router (https port 4566)...
localstack_1  | Starting mock DynamoDB service on http port 4566 ...
localstack_1  | Starting mock DynamoDB Streams service on http port 4566 ...
localstack_1  | Starting mock Kinesis service on http port 4566 ...
localstack_1  | Starting mock SQS service on http port 4566 ...
localstack_1  | qemu: uncaught target signal 11 (Segmentation fault) - core dumped
localstack_1  | 2021-10-16T18:47:59:DEBUG:localstack.utils.common: Starting download from https://github.com/localstack/localstack-artifacts/raw/master/local-certs/server.key to /tmp/localstack/server.test.pem (5159 bytes)
localstack_1  | 2021-10-16T18:47:59:DEBUG:localstack.utils.common: Done downloading https://github.com/localstack/localstack-artifacts/raw/master/local-certs/server.key, response code 200, total bytes 7571
localstack_1  | 2021-10-16T18:47:59:DEBUG:localstack.utils.run: Executing command: cd "/opt/code/localstack/localstack/infra/dynamodb" && zip -u DynamoDBLocal.jar log4j2.xml || true
localstack_1  | [2021-10-16 18:47:59 +0000] [51] [INFO] Running on https://0.0.0.0:4566 (CTRL + C to quit)
localstack_1  | 2021-10-16T18:47:59:INFO:hypercorn.error: Running on https://0.0.0.0:4566 (CTRL + C to quit)
localstack_1  | 2021-10-16T18:47:59:DEBUG:localstack.utils.run: Executing command: cd /opt/code/localstack/localstack/infra/dynamodb/; java -Djava.library.path=./DynamoDBLocal_lib -Xmx256m -jar DynamoDBLocal.jar -port 54527 -inMemory
localstack_1  | 2021-10-16 18:47:59,563:API:  * Running on all addresses.
localstack_1  |    WARNING: This is a development server. Do not use it in a production deployment.
localstack_1  | 2021-10-16T18:47:59:DEBUG:localstack.services.install: getting kinesis-mock for linux x86_64
localstack_1  | 2021-10-16T18:47:59:DEBUG:localstack.services.install: kinesis-mock found at /opt/code/localstack/localstack/infra/kinesis-mock/kinesis-mock-linux-amd64-static
localstack_1  | 2021-10-16T18:47:59:DEBUG:localstack.utils.run: Executing command: KINESIS_MOCK_PLAIN_PORT=52959 SHARD_LIMIT=100 CREATE_STREAM_DURATION=500ms DELETE_STREAM_DURATION=500ms REGISTER_STREAM_CONSUMER_DURATION=500ms START_STREAM_ENCRYPTION_DURATION=500ms STOP_STREAM_ENCRYPTION_DURATION=500ms DEREGISTER_STREAM_CONSUMER_DURATION=500ms MERGE_SHARDS_DURATION=500ms SPLIT_SHARD_DURATION=500ms UPDATE_SHARD_COUNT_DURATION=500ms  LOG_LEVEL=INFO  /opt/code/localstack/localstack/infra/kinesis-mock/kinesis-mock-linux-amd64-static --gc=G1
localstack_1  | 2021-10-16T18:47:59:INFO:localstack.multiserver: Starting multi API server process on port 46937
localstack_1  | 2021-10-16T18:47:59:ERROR:localstack.services.kinesis.kinesis_starter: kinesis terminated with return code -11
localstack_1  | [2021-10-16 18:47:59 +0000] [51] [INFO] Running on http://0.0.0.0:46937 (CTRL + C to quit)
localstack_1  | 2021-10-16T18:47:59:INFO:hypercorn.error: Running on http://0.0.0.0:46937 (CTRL + C to quit)
localstack_1  | 2021-10-16 18:47:59,710:API:  * Running on all addresses.
localstack_1  |    WARNING: This is a development server. Do not use it in a production deployment.
localstack_1  | Initializing DynamoDB Local with the following configuration:
localstack_1  | Port:   54527
localstack_1  | InMemory:   true
localstack_1  | DbPath: null
localstack_1  | SharedDb:   false
localstack_1  | shouldDelayTransientStatuses:   false
localstack_1  | CorsParams: *
localstack_1  |
localstack_1  | Waiting for all LocalStack services to be ready
localstack_1  | Waiting for all LocalStack services to be ready
localstack_1  | 2021-10-16T18:48:15:WARNING:localstack.services.plugins: Service "dynamodb" not yet available, retrying...
localstack_1  | Waiting for all LocalStack services to be ready
localstack_1  | Waiting for all LocalStack services to be ready
chezsmithy commented 2 years ago

Likely resolved by: https://github.com/localstack/localstack/issues/2550

thrau commented 2 years ago

hi @viswanath7 @caiorcferreira, 0.13 includes a native ARM64 image that hopefully resolves your issue. could you give it a try?

Herraj commented 2 years ago

This is fixed by #2550. Just need to pull the latest docker images

levpaul commented 2 years ago

@Herraj - still seeing this in 0.14.4.dev

Image info:

                "LOCALSTACK_BUILD_DATE=2022-06-22",
                "LOCALSTACK_BUILD_GIT_HASH=586bdbb4",
                "LOCALSTACK_BUILD_VERSION=0.14.4.dev",

Basically I'm seeing this segfault when trying to use kinesis on an AMD64 localstack image on an M1. All other services still work fine (sqs, awssm, etc)

/usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initaws.d/10-kinesis-stuff.sh
2022-06-22T23:27:47.117:DEBUG:plugin.manager: instantiating plugin PluginSpec(localstack.aws.provider.sqs:default = <function sqs at 0x4005c84940>)
2022-06-22T23:27:47.118:DEBUG:plugin.manager: loading plugin localstack.aws.provider:sqs:default
2022-06-22T23:27:47.164:DEBUG:localstack.services.plugins: checking service health sqs:4566
2022-06-22T23:27:47.170:DEBUG:localstack.services.sqs.provider: creating queue key=BigQueue attributes=None tags=None
MER_DURATION': '500ms', 'MERGE_SHARDS_DURATION': '500ms', 'SPLIT_SHARD_DURATION': '500ms', 'UPDATE_SHARD_COUNT_DURATION': '500ms', 'LOG_LEVEL': 'INFO'}
2022-06-22T23:27:49.258:DEBUG:localstack.utils.run: Executing command: ['/opt/code/localstack/localstack/infra/kinesis-mock/kinesis-mock-linux-amd64-static', '--gc=G1']
2022-06-22T23:27:49.259:INFO:localstack.services.infra: Starting mock Kinesis service on http port 4566 ...
2022-06-22T23:27:49.292:INFO:localstack.services.kinesis.kinesis_mock_server: qemu: uncaught target signal 11 (Segmentation fault) - core dumped
2022-06-22T23:27:49.293:INFO:localstack.utils.run: Restarting process (received exit code -11): ['/opt/code/localstack/localstack/infra/kinesis-mock/kinesis-mock-linux-amd64-static', '--gc=G1']
2022-06-22T23:27:49.293:DEBUG:localstack.utils.run: Executing command: ['/opt/code/localstack/localstack/infra/kinesis-mock/kinesis-mock-linux-amd64-static', '--gc=G1']
2022-06-22T23:27:49.311:INFO:localstack.services.kinesis.kinesis_mock_server: qemMER_DURATION': '500ms', 'MERGE_SHARDS_DURATION': '500ms', 'SPLIT_SHARD_DURATION': '500ms', 'UPDATE_SHARD_COUNT_DURATION': '500ms', 'LOG_LEVEL': 'INFO'}
2022-06-22T23:27:49.258:DEBUG:localstack.utils.run: Executing command: ['/opt/code/localstack/localstack/infra/kinesis-mock/kinesis-mock-linux-amd64-static', '--gc=G1']
2022-06-22T23:27:49.259:INFO:localstack.services.infra: Starting mock Kinesis service on http port 4566 ...
2022-06-22T23:27:49.292:INFO:localstack.services.kinesis.kinesis_mock_server: qemu: uncaught target signal 11 (Segmentation fault) - core dumped
2022-06-22T23:27:49.293:INFO:localstack.utils.run: Restarting process (received exit code -11): ['/opt/code/localstack/localstack/infra/kinesis-mock/kinesis-mock-linux-amd64-static', '--gc=G1']
2022-06-22T23:27:49.293:DEBUG:localstack.utils.run: Executing command: ['/opt/code/localstack/localstack/infra/kinesis-mock/kinesis-mock-linux-amd64-static', '--gc=G1']
2022-06-22T23:27:49.311:INFO:localstack.services.kinesis.kinesis_mock_server: qemu: uncaught target signal 11 (Segmentation fault) - core dumped
2022-06-22T23:27:49.311:INFO:localstack.utils.run: Restarting process (received exit code -11): ['/opt/code/localstack/localstack/infra/kinesis-mock/kinesis-mock-linux-amd64-static', '--gc=G1']
2022-06-22T23:27:49.312:DEBUG:localstack.utils.run: Executing command: ['/opt/code/localstack/localstack/infra/kinesis-mock/kinesis-mock-linux-amd64-static', '--gc=G1']
2022-06-22T23:27:49.327:INFO:localstack.services.kinesis.kinesis_mock_server: qemu: uncaught target signal 11 (Segmentation fault) - core dumped
2022-06-22T23:27:49.328:INFO:localstack.utils.run: Restarting process (received exit code -11): ['/opt/code/localstack/localstack/infra/kinesis-mock/kinesis-mock-linux-amd64-static', '--gc=G1']
2022-06-22T23:27:49.329:DEBUG:localstack.utils.run: Executing command: ['/opt/code/localstack/localstack/infra/kinesis-mock/kinesis-mock-linux-amd64-static', '--gc=G1']
neverchanje commented 1 year ago

I still encountered this problem on my M1 Mac:

docker run -p 4567:4567 -p 4568:4568 ghcr.io/etspaceman/kinesis-mock:0.3.10

WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested docker: Error response from daemon: driver failed programming external connectivity on endpoint beautiful_rubin (c2305baf8f5c0f5202fab006abaa3487431cff0d43fafc995fc53b30ae591875): Bind for 0.0.0.0:4568 failed: port is already allocated.