Failed to verify ink contracts

[saki-osive]

I followed the tutorial provided here for verifying ink smart contracts. However, I'm facing an error:

The script in the ink-verifier-image repo under scripts/verify-contract is failing at line number 26:

..
mkdir -p $TARGET
..

Is there a step missing in the tutorial?

[XY-Wang]

Hi,

it seems like the container does not have the permission to create the directory build/package.

This is how it works:

1. Ink verifier server runs in its own process, with the user that executes the command.
2. This process creates a processing directory in the host file system that will have the default permissions according to the umask of your system. By default, the processing directory is located at ./.tmp/processing/<network_id>/<code_hash>.
3. When a verification process is triggered it will run a docker container mapping the processing directory to the /build directory inside the container.
4. If the container is not running in rootless mode or the OS is not SELinux it should work with the default configuration, since the container uses uid 0.

Note that we tested it out in rootless docker and with podman, is a more secure setup but it needs extra configuration.

Could you please provide the output to the command docker info?

[saki-osive]

@XY-Wang @mfornos Sure.

Here is the output of the docker info command:

arcturus@arcturus:~$ docker info
Client:
 Context:    default
 Debug Mode: false

Server:
 Containers: 10
  Running: 1
  Paused: 0
  Stopped: 9
 Images: 62
 Server Version: 20.10.16
 Storage Driver: zfs
  Zpool: rpool
  Zpool Health: ONLINE
  Parent Dataset: rpool/var/lib/docker
  Space Used By Parent: 8162181120
  Space Available: 389698441216
  Parent Quota: no
  Compression: lz4
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 
 runc version: 
 init version: 
 Security Options:
  apparmor
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.19.0-28-generic
 Operating System: Ubuntu 22.10
 OSType: linux
 Architecture: x86_64
 CPUs: 32
 Total Memory: 62.71GiB
 Name: arcturus
 ID: NS34:HLO5:ULJB:EG2X:XRCG:YC3R:OHMX:C44K:TXNM:BO6N:UH37:2MQ6
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Does it provide any help? Thanks.

[XY-Wang]

Hmm... did you happen to be able to build a verifiable package with the command line tool according to this part of the tutorial? If you haven't, could you please try it to see if you get similar errors?

Could you also provide the output of an ls -la command in the [path]/ink-verifier-server/.tmp/ directory?

[saki-osive]

@XY-Wang Thanks for the reply.

Yes, the command line tool named build-verifiable-ink was built successfully and is available as a cargo binary:

The verifiable package was successfully built using:

..

Generate the verifiable source code package

build-verifiable-ink -t develop . ..

Here is the output of the [path]/ink-verifier-server/.tmp directory:

[XY-Wang]

Could you please try to remove the staging and processing folders in .tmp/ and restart the server? Do you still run into the same problem?

Are you able to do a docker container inspect on the verification container while it's running?

[XY-Wang]

Hi @saki-osive, did you manage to solve the issue?

[saki-osive]

Could you please try to remove the staging and processing folders in .tmp/ and restart the server? Do you still run into the same problem?

Are you able to do a docker container inspect on the verification container while it's running?

@XY-Wang I tried these steps but still, I'm getting the same error.

[XY-Wang]

@saki-osive could you paste the output of the following commands?

1. docker run -it --privileged --entrypoint id ink-verifier:develop
2. docker run -it --entrypoint ls -v <PATH_TO_YOUR_TARGET_DIR>:/build ink-verifier:develop -la .

[saki-osive]

Sure @XY-Wang

1.

Running

arcturus@pop-os:~/patch/ink_verification/ink-verifier-image$ docker run -it --privileged --entrypoint id ink-verifier:develop

outputs

uid=0(root) gid=0(root) groups=0(root)

2.

Running

arcturus@pop-os:~/patch/ink_verification/dev-contracts-substrate/example-contracts/flipper/target$ docker run -it --entrypoint ls -v /home/arcturus/patch/ink_verification/dev-contracts-substrate/example-contracts/flipper/target:/build ink-verifier:develop -la .

outputs

total 12
drwxr-xr-x  3 root root 4096 Jan 23 16:36 .
drwxr-xr-x 23 root root 4096 Jan 23 19:40 ..
drwxr-xr-x  3 root root 4096 Jan 23 16:47 ink

[mfornos]

Just to confirm, you didn't modified the apparmor default docker profile and the create directory is not being forbidden by apparmor (demsg | grep apparmor | grep docker to check)?

On Mon, 23 Jan 2023, 20:45 saki-osive, @.***> wrote:

Sure @XY-Wang https://github.com/XY-Wang

1.

Running

@.***:~/patch/ink_verification/ink-verifier-image$ docker run -it --privileged --entrypoint id ink-verifier:develop

outputs

uid=0(root) gid=0(root) groups=0(root)

1.

Running

@.***:~/patch/ink_verification/dev-contracts-substrate/example-contracts/flipper/target$ docker run -it --entrypoint ls -v /home/arcturus/patch/ink_verification/dev-contracts-substrate/example-contracts/flipper/target:/build ink-verifier:develop -la .

outputs

total 12 drwxr-xr-x 3 root root 4096 Jan 23 16:36 . drwxr-xr-x 23 root root 4096 Jan 23 19:40 .. drwxr-xr-x 3 root root 4096 Jan 23 16:47 ink

— Reply to this email directly, view it on GitHub https://github.com/web3labs/ink-verifier-image/issues/1#issuecomment-1400884520, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFCCICOBW7MGCREBTY3VBDWT3NTZANCNFSM6AAAAAATTESXHU . You are receiving this because you were mentioned.Message ID: @.***>

[saki-osive]

@mfornos Ummm.. Although a little unclear, I assume you're asking if I modified the docker profile.

No, I didn't.

[mfornos]

So, if you run docker run -v <PATH_TO_YOUR_TARGET_DIR>:/build ink-verifier:develop the verification gives you a different error?

[saki-osive]

So, if you run docker run -v <PATH_TO_YOUR_TARGET_DIR>:/build ink-verifier:develop the verification gives you a different error?

Hi @mfornos , running this gives me the output: No archive found

[mfornos]

We will try to figure out if the problem is related with the default parameters that ink verifier server uses to run the container.

Please, try the following commands:

1. docker run -it --entrypoint mkdir -v <PATH_TO_YOUR_TARGET_DIR>:/build ink-verifier:develop -v a
2. docker run -it --security-opt=no-new-privileges --cap-drop all --entrypoint mkdir -v <PATH_TO_YOUR_TARGET_DIR>:/build ink-verifier:develop -v b

[saki-osive]

@mfornos Sure,

1. Running

arcturus@pop-os:~/patch/ink_verification/dev-contracts-substrate/example-contracts/flipper/target$ docker run -it --entrypoint mkdir -v /home/arcturus/patch/ink_verification/dev-contracts-substrate/example-contracts/flipper/target:/build ink-verifier:develop -v a

outputs

mkdir: created directory 'a'

2. Running

arcturus@pop-os:~/patch/ink_verification/dev-contracts-substrate/example-contracts/flipper/target$ docker run -it --security-opt=no-new-privileges --cap-drop all --entrypoint mkdir -v /home/arcturus/patch/ink_verification/dev-contracts-substrate/example-contracts/flipper/target:/build ink-verifier:develop -v b

outputs

mkdir: created directory 'b'

[mfornos]

The permission problem is not reproducible with the docker run, let's check if it is related with node.js.

Please, save the attached test_docker_run.zip script, unzip and execute it: node test_docker_run.js /home/arcturus/patch/ink_verification/dev-contracts-substrate/example-contracts/flipper/target/test

[saki-osive]

Sure @mfornos,

Running

arcturus@pop-os:~/patch/ink_verification/dev-contracts-substrate/example-contracts/flipper/target$ node test_docker_run.js /home/arcturus/patch/ink_verification/dev-contracts-substrate/example-contracts/flipper/target/test

outputs

Spawn Docker params: [
  'run',
  '--rm',
  '--security-opt=no-new-privileges',
  '--cap-drop',
  'all',
  '-v',
  '/home/arcturus/patch/ink_verification/dev-contracts-substrate/example-contracts/flipper/target/test:/build',
  '--entrypoint',
  'mkdir',
  'ink-verifier:develop',
  '-v',
  'xyz'
] 
---

User Info: {
  uid: 1000,
  gid: 1000,
  username: 'arcturus',
  homedir: '/home/arcturus',
  shell: '/bin/bash'
} 
---

mkdir: created directory 'xyz'
248170 exit 0

[XY-Wang]

Hmmm this is really strange since the script mimics the process of ink-verifier-server i.e. start a node process which spawns a docker container which tries to create a directory We expected that you'd run into the same error but clearly that's not the case...

From the test docker runs it seems like your docker is running with root access so in theory there shouldn't be any permission issues.

The only thing we can see here, for the moment, is that in the earlier screenshots you were running commands from arcturus@arcturus and in the most recent successful runs it has changed to arcturus@pop-os. Are you running in a different host or has your machine set up been changed? If you run the server from arcturus@pop-os do you still get the same permission issue?

[saki-osive]

Hi @XY-Wang Thanks for the reply.

The set-up didn't seem to work on either of the machines, pop-os or arcturus is just the hostname, which can be changed as well.

[mfornos]

:thinking: As @XY-Wang said the server does the same as the troubleshooting script we provided.

There is no apparent logical reason for it to fail creating the directory and we cannot reproduce the permission issue, not in our development machines, not in the servers (running other OS), not even in your own host with the provided script.

Do you have an AppArmor profile for the npm command? You could check your profiles in /etc/apparmor.d/ and see the ones being applied executing sudo aa-status

[saki-osive]

Sure @mfornos,

1. Running

   arcturus@pop-os:/etc/apparmor.d$ ls /etc/apparmor.d/

   outputs

   abi             local            tunables         usr.lib.libreoffice.program.oosplash     usr.sbin.chronyd
   abstractions    lsb_release      usr.bin.evince   usr.lib.libreoffice.program.senddoc      usr.sbin.cups-browsed
   disable         nvidia_modprobe  usr.bin.man      usr.lib.libreoffice.program.soffice.bin  usr.sbin.cupsd
   force-complain  sbin.dhclient    usr.bin.tcpdump  usr.lib.libreoffice.program.xpdfimport   usr.sbin.rsyslogd

2. Running

   arcturus@pop-os:/etc/apparmor.d$ sudo aa-status

   outputs

   
   apparmor module is loaded.
   27 profiles are loaded.
   25 profiles are in enforce mode.
   /usr/bin/evince
   /usr/bin/evince-previewer
   /usr/bin/evince-previewer//sanitized_helper
   /usr/bin/evince-thumbnailer
   /usr/bin/evince//sanitized_helper
   /usr/bin/man
   /usr/lib/NetworkManager/nm-dhcp-client.action
   /usr/lib/NetworkManager/nm-dhcp-helper
   /usr/lib/connman/scripts/dhclient-script
   /usr/lib/cups/backend/cups-pdf
   /usr/sbin/chronyd
   /usr/sbin/cups-browsed
   /usr/sbin/cupsd
   /usr/sbin/cupsd//third_party
   /{,usr/}sbin/dhclient
   docker-default
   libreoffice-senddoc
   libreoffice-soffice//gpg
   libreoffice-xpdfimport
   lsb_release
   man_filter
   man_groff
   nvidia_modprobe
   nvidia_modprobe//kmod
   tcpdump
   2 profiles are in complain mode.
   libreoffice-oosplash
   libreoffice-soffice
   0 profiles are in kill mode.
   0 profiles are in unconfined mode.
   58 processes have profiles defined.
   58 processes are in enforce mode.
   /usr/sbin/chronyd (1240) 
   /usr/sbin/chronyd (1244) 
   /usr/sbin/cups-browsed (237056) 
   /usr/sbin/cupsd (237027) 
   /usr/local/bin/node (25663) docker-default
   /usr/local/bin/node (26227) docker-default
   /usr/lib/postgresql/15/bin/postgres (26543) docker-default
   /usr/local/bin/substrate-contracts-node (26550) docker-default
   /usr/sbin/nginx (26748) docker-default
   /usr/lib/postgresql/15/bin/postgres (26934) docker-default
   /usr/local/bin/node (27317) docker-default
   /usr/lib/postgresql/15/bin/postgres (27330) docker-default
   /usr/lib/postgresql/15/bin/postgres (27331) docker-default
   /usr/lib/postgresql/15/bin/postgres (27451) docker-default
   /usr/lib/postgresql/15/bin/postgres (27452) docker-default
   /usr/lib/postgresql/15/bin/postgres (27453) docker-default
   /usr/sbin/nginx (27612) docker-default
   /usr/sbin/nginx (27613) docker-default
   /usr/sbin/nginx (27614) docker-default
   /usr/sbin/nginx (27615) docker-default
   /usr/sbin/nginx (27617) docker-default
   /usr/sbin/nginx (27618) docker-default
   /usr/sbin/nginx (27619) docker-default
   /usr/sbin/nginx (27620) docker-default
   /usr/sbin/nginx (27621) docker-default
   /usr/sbin/nginx (27622) docker-default
   /usr/sbin/nginx (27623) docker-default
   /usr/sbin/nginx (27624) docker-default
   /usr/sbin/nginx (27625) docker-default
   /usr/sbin/nginx (27626) docker-default
   /usr/sbin/nginx (27627) docker-default
   /usr/sbin/nginx (27628) docker-default
   /usr/sbin/nginx (27629) docker-default
   /usr/sbin/nginx (27630) docker-default
   /usr/sbin/nginx (27631) docker-default
   /usr/sbin/nginx (27632) docker-default
   /usr/sbin/nginx (27633) docker-default
   /usr/sbin/nginx (27634) docker-default
   /usr/sbin/nginx (27635) docker-default
   /usr/sbin/nginx (27636) docker-default
   /usr/sbin/nginx (27637) docker-default
   /usr/sbin/nginx (27638) docker-default
   /usr/sbin/nginx (27639) docker-default
   /usr/sbin/nginx (27640) docker-default
   /usr/sbin/nginx (27641) docker-default
   /usr/sbin/nginx (27642) docker-default
   /usr/sbin/nginx (27643) docker-default
   /usr/sbin/nginx (27644) docker-default
   /usr/lib/postgresql/15/bin/postgres (27742) docker-default
   /usr/lib/postgresql/15/bin/postgres (27743) docker-default
   /usr/lib/postgresql/15/bin/postgres (27745) docker-default
   /usr/lib/postgresql/15/bin/postgres (27746) docker-default
   /usr/lib/postgresql/15/bin/postgres (27747) docker-default
   /substrate-gateway/substrate-gateway (27749) docker-default
   /usr/local/bin/node (27866) docker-default
   /usr/local/bin/node (28806) docker-default
   /usr/lib/postgresql/15/bin/postgres (28928) docker-default
   /usr/lib/postgresql/15/bin/postgres (688933) docker-default
   0 processes are in complain mode.
   0 processes are unconfined but have a profile defined.
   0 processes are in mixed mode.
   0 processes are in kill mode.

[saki-osive]

🤔 As @XY-Wang said the server does the same as the troubleshooting script we provided.

There is no apparent logical reason for it to fail creating the directory and we cannot reproduce the permission issue, not in our development machines, not in the servers (running other OS), not even in your own host with the provided script.

Do you have an AppArmor profile for the npm command? You could check your profiles in /etc/apparmor.d/ and see the ones being applied executing sudo aa-status

Well I have tried it 10+ times on 4+ systems with different operating systems. I guess there's some step missing from the tutorial.

@XY-Wang @mfornos

[mfornos]

Hi @saki-osive

Finally we reproduced the issue! :frog:

It was caused by the "--cap-drop all" security flag in the docker run command. Nevertheless you will still have other problems related to running the container mapped to the root user in the host.

So, to avoid more issues, we have decided to update the tutorial with a prerequisite configuration of the docker daemon (i.e. user namespace remapping to the host user id).

Check it here, please: https://github.com/web3labs/ink-verifier-server/blob/main/docs/TUTORIAL.md#docker-user-namespace-remapping

[saki-osive]

Hi @mfornos , thank you for the reply.

I re-tested the updated doc.

However, I'm facing this issue:

Any idea how this can be fixed?

@XY-Wang

[mfornos]

We have added many changes in the system, so you need to re-deploy everything from scratch.

On Thu, 9 Feb 2023, 07:32 saki-osive, @.***> wrote:

Hi @mfornos https://github.com/mfornos , thank you for the reply.

I re-tested the updated doc.

However, I'm facing this issue:

[image: image] https://user-images.githubusercontent.com/62865682/217735498-81f46622-fe76-4ba4-95e3-4b222ab94480.png

Any idea how this can be fixed?

@XY-Wang https://github.com/XY-Wang

— Reply to this email directly, view it on GitHub https://github.com/web3labs/ink-verifier-image/issues/1#issuecomment-1423704323, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFCCIDBSLQWIYYVJVJ7WILWWSFODANCNFSM6AAAAAATTESXHU . You are receiving this because you were mentioned.Message ID: @.***>

[XY-Wang]

Hi @saki-osive ,

Please do a git pull on all the repos since we've made many changes :smiley: Also, since you've restarted docker all your existing docker images would have been wiped, so you'll need to run docker build . -t ink-verifier:develop again.

We've amended in the tutorial the command to run ink-verifier-server to include setting the server host to 0.0.0.0 so that it can be accessed from the docker container. Check it out here.

Please also note that if you're running the docker compose files in epirus-substrate/local-testnet/ separately (i.e. docker-compose -f docker-compose.{SERVICE}.yml up -d) instead of using the run-all.sh script, you'll need to do a docker-compose -f docker-compose.{SERVICE}.yml pull to get the latest image. If you run the run-all.sh script the pull is already taken care of.

Hope that helps!

[saki-osive]

Hi @mfornos @XY-Wang , thank you for the reply.

I removed all directories, docker images, volumes and containers. After that I cloned all repos and started everything from scratch.

Steps I followed:

• Ran ./run-all.sh under epirus-substrate/local-testnet : RAN SUCCESSFULLY

• npm install and SERVER_HOST=0.0.0.0 npm start under ink-verifier-server: RAN SUCCESSFULLY

• Ran docker build . -t ink-verifier:develop under ink-verifier-image: RAN SUCCESSFULLY

• Ran cargo install --path . under cli/: RAN SUCCESSFULLY and built the binary

• Ran build-verifiable-ink -t develop . under dev-contracts-substrate/example-contracts/flipper/: RAN SUCCESSFULLY and outputed the contract files

• Then I uploaded the contract on this UI.

• The contract was uploaded successfully but it kept loading during calling it.

• Finally, I was able to make 3-4 calls on the flipper and bool functions to reverse the true and false values after reloading the UI 2-3 times.

After this, I went to localhost:3000, but it again showed me like this:

The error message is this:

I have tested this 3-4 times on different systems. I believe something is broken/not documented?

[XY-Wang]

Hmm... could you please paste the logs of the ./run-all.sh when this error happens?

[saki-osive]

Hi @XY-Wang, I stopped all the running containers and invoked the command again and it worked. But still doesn't work on the first attempt, maybe it needs reloading?

[XY-Wang]

That's great! The issue with the first run is probably that your browser had the old version cached :smile:

[saki-osive]

That's great! The issue with the first run is probably that your browser had the old version cached smile

Can't confirm it's a browser cache issue as I tried in Incognito mode and another browser too.