Failure to start (Docker / Kubernetes)

[Zorlin]

Hi there,

We are attempting to run Motion as a pod inside Kubernetes, using the standard Docker image provided at ghcr.io/filecoin-project/motion:main.

We're running it against a working Singularity instance, also running in k8s:

command:
        - motion
        - '--localWalletDir=/storage/wallet'
        - '--experimentalSingularityStore'
        - '--experimentalRemoteSingularityAPIUrl=http://singularity-api:9090'
      envFrom:
        - configMapRef:
            name: test-motion-config

with the environment config:

MOTION_STORE_DIR: /storage/singularity

Singularity is running and made available at http://singularity-api:9090 by Kubernetes. Attempting to start the Motion pod results in the following:

2023-08-30T14:56:08.756Z    INFO    motion/wallet   wallet/disk_keystore.go:39  Using specified disk keystore path  {"path": "/storage/wallet"}
2023-08-30T14:56:08.757Z    INFO    motion/wallet   wallet/disk_keystore.go:72  Opened disk keystore successfully   {"path": "/storage/wallet"}
2023-08-30T14:56:08.758Z    INFO    motion/wallet   wallet/wallet.go:54 Found 1 wallet addresses
2023-08-30T14:56:08.758Z    INFO    motion/wallet   wallet/wallet.go:55 Wallet addresses    {"addresses": ["f1e7g7tcwxjqd5exz6t764nb6w67ju62cwygk2svq"]}
2023-08-30T14:56:08.759Z    INFO    motion/cmd  motion/main.go:144  Using Singularity blob store    {"storeDir": "/storage/singularity"}
2023-08-30T14:56:08.767Z    ERROR   motion/cmd  motion/main.go:146  Failed to start Singularity blob store  {"err": "invalid parameter: failed to parse scanningState: invalid work state"}
2023-08-30T14:56:08.767Z    ERROR   motion/cmd  motion/main.go:196  invalid parameter: failed to parse scanningState: invalid work state

[hannahhoward]

@Zorlin when did you run this? can you confirm your motion main is up to date? If not can you re-pull. This bug did exist but it should be resolved in the latest version of main.

[hannahhoward]

(or in the Kubernetes case, set your image pull policy to always pull)

[hannahhoward]

aaannnd I realized we actually have a different problem in main.

BUT, I'm curious if you get that other problem, which will be a wallet error that says "invalid private key"

[Zorlin]

This was about... 3 days ago? I think?

I'll try re-pulling from main and see what we produce.

Thanks!

On Fri, Sep 1, 2023, 2:11 AM Hannah Howard @.***> wrote:

aaannnd I realized we actually have a different problem in main.

BUT, I'm curious if you get that other problem, which will be a wallet error that says "invalid private key"

— Reply to this email directly, view it on GitHub https://github.com/filecoin-project/motion/issues/79#issuecomment-1701533276, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKONLCUMMAX42BIYVUOZRDXYDHTTANCNFSM6AAAAAA4EWIFAE . You are receiving this because you were mentioned.Message ID: @.***>

[celine3650]

Hey @Zorlin , I am following up on this. Did you get the same error after re-pulling?

[Zorlin]

Hey @Zorlin , I am following up on this. Did you get the same error after re-pulling?

Hi @celine3650 - different error came up earlier...

2023-09-05T19:53:55.242Z    INFO    motion/wallet   wallet/disk_keystore.go:39  Using specified disk keystore path  {"path": "/storage/wallet"}
2023-09-05T19:53:55.242Z    INFO    motion/wallet   wallet/disk_keystore.go:72  Opened disk keystore successfully   {"path": "/storage/wallet"}
2023-09-05T19:53:55.250140633Z 2023-09-05T19:53:55.243Z INFO    motion/wallet   wallet/wallet.go:54 Found 1 wallet addresses
2023-09-05T19:53:55.243Z    INFO    motion/wallet   wallet/wallet.go:55 Wallet addresses    {"addresses": ["REDACTED"]}
2023-09-05T19:53:55.243Z    INFO    motion/cmd  motion/main.go:218  Using Singularity blob store    {"storeDir": "/storage/singularity"}
2023-09-05T19:53:55.251Z    ERROR   motion/cmd  motion/main.go:220  Failed to start Singularity blob store  {"err": "failed to check source: directory not found", "errVerbose": "failed to check source: directory not found\ngithub.com/data-preservation-programs/singularity/client/http.parseHTTPError\n\t/go/pkg/mod/github.com/data-preservation-programs/singularity@v0.3.1-0.20230901000338-bb3ac7e94619/client/http/client.go:375\ngithub.com/data-preservation-programs/singularity/client/http.(*Client).CreateLocalSource\n\t/go/pkg/mod/github.com/data-preservation-programs/singularity@v0.3.1-0.20230901000338-bb3ac7e94619/client/http/client.go:75\ngithub.com/filecoin-project/motion/integration/singularity.(*SingularityStore).Start\n\t/go/src/motion/integration/singularity/store.go:81\nmain.main.func1\n\t/go/src/motion/cmd/motion/main.go:219\ngithub.com/urfave/cli/v2.(*Command).Run\n\t/go/pkg/mod/github.com/urfave/cli/v2@v2.25.7/command.go:274\ngithub.com/urfave/cli/v2.(*App).RunContext\n\t/go/pkg/mod/github.com/urfave/cli/v2@v2.25.7/app.go:332\ngithub.com/urfave/cli/v2.(*App).Run\n\t/go/pkg/mod/github.com/urfave/cli/v2@v2.25.7/app.go:309\nmain.main\n\t/go/src/motion/cmd/motion/main.go:249\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1598"}
2023-09-05T19:53:55.251310366Z 2023-09-05T19:53:55.251Z ERROR   motion/cmd  motion/main.go:250  failed to check source: directory not found

However, a bit of hacking later (attempting to get latest :main for Singularity) I am now completely unable to run Singularity thanks to an exec format error caused by (in theory) a container build misconfiguration... https://github.com/data-preservation-programs/singularity/issues/270 - will check back here once that one gets fixed. :)

[hannahhoward]

@Zorlin

Ok, in an effort to keep things simple, we've cut version v0.2.0 that we've tested and made deals with mainnet on. We're still in rapid development and you may encounter issues if you use the main branch in your testing.

We've also fixed the build issue with singularity.

It's worth mentioning that motion + singularity require several processes to be running now. My recommendation for your kubernetes work is to study the docker-compose.yml file along with the .env.example settings and the instructions in the README. You should be able to translate the overall docker compose setup to on in kubernetes. We can also answer questions if you get stuck along the way.

I'm going to close this issue for now but feel free to reach out if you encounter additional issues in setting up kubernetes.

[Zorlin]

Thanks Hannah, will do! Sorry for the delays in responses, prepping for FIL Dev SG

On Sat, Sep 9, 2023, 4:08 AM Hannah Howard @.***> wrote:

Closed #79 https://github.com/filecoin-project/motion/issues/79 as completed.

— Reply to this email directly, view it on GitHub https://github.com/filecoin-project/motion/issues/79#event-10321795511, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKONLDJPXMPT6HD7HXFQNTXZN3MXANCNFSM6AAAAAA4EWIFAE . You are receiving this because you were mentioned.Message ID: @.***>

[celine3650]

@Zorlin Could you post the error you mentioned in the DM as of Sep 19?

[xinaxu]

outdated