type: ACQUIRE_ERR reason: user_install_error

[rippleitinnz]

Recently I had somebody try to set up 3 contracts on my host - all failed with the

type: ACQUIRE_ERR reason: user_install_error

error.

is this an error with my host or theirs. and if mine, what are the possible causes.

From my POV everything is running just fine

tx ID 918CE198E6FE9262E9473C681BE2CCD969CFD71C8092426CEFD1517497DDA398

[chalith]

You can check evernode log and see if there are any errors while the instance creation. This shows last 200 lines of logs in sashimono and message board.

If the particular logs are older than 200 lines you can run

journalctl -u sashimono-agent.service
sudo -u sashimbxrpl bash -c journalctl --user -u sashimono-mb-xrpl

And check the logs

You can send the logs here and we can investigate the error

[rippleitinnz]

Thanks chalith

Here's the relevant section from my log

https://pastebin.com/PtSCqNCm

[BimsaraFernando]

Hi @rippleitinnz, It seems instance creation failed with an error when creating the file system for the docker container. Shall we try again after restarting the VPS?

[rippleitinnz]

I've restarted - it's locally hosted so you can retry.

[chalith]

Thank you. We've tried acquiring again. It seems the same error. Can you send us the logs again

[rippleitinnz]

Thank you. We've tried acquiring again. It seems the same error. Can you send us the logs again

Sure.

Here we go.

https://pastebin.com/PZzQfpUt

[chalith]

Thank you. Seems this log got more info than the previous. Let us do more investigation and come back to you.

[chalith]

@rippleitinnz can you check cat /etc/fstab and check these properties exist grpjquota=aquota.group,jqfmt=vfsv0

[rippleitinnz]

@chalith

No they don't.

[chalith]

Okay, The issue could be due to that reason. Can you try adding grpjquota=aquota.group,jqfmt=vfsv0 into the options of the line which containing root mount / in /etc/fstab And then running mount -o remount / As a precautionary measure make sure to keep a backup of /etc/fstab and your host secret key before doing this.

[rippleitinnz]

Okay, The issue could be due to that reason. Can you try adding grpjquota=aquota.group,jqfmt=vfsv0 into the options of the line which containing root mount / in /etc/fstab And then running mount -o remount / As a precautionary measure make sure to keep a backup of /etc/fstab and your host secret key before doing this.

Given that it is a ZFS file system I don't think fstab manages my root filesystem, I think ZFS does

/dev/nvme0n1p2: UUID="8ad46f51-aa66-4446-825e-8037a8e6a2c7" TYPE="swap" PARTUUID="72cb4138-a162-f942-8729-c7374e0e131d"
/dev/nvme0n1p1: UUID="7301-CC6E" TYPE="vfat" PARTLABEL="EFI System Partition" PARTUUID="dbd701fb-5fa2-4bd5-a6e4-e9079104f3fd"
/dev/nvme0n1p3: LABEL="bpool" UUID="18053383433121191081" UUID_SUB="16030335581014003639" TYPE="zfs_member" PARTUUID="bfbed895-c84e-e34c-85f0-73d7996b7f85"
/dev/nvme0n1p4: LABEL="rpool" UUID="14666425284636280899" UUID_SUB="2673239875637231341" TYPE="zfs_member" PARTUUID="0d033d33-a628-a442-87c8-0cec0748cea3"
/dev/loop0: TYPE="squashfs"
/dev/loop2: TYPE="squashfs"
/dev/loop3: TYPE="squashfs"
/dev/loop4: TYPE="squashfs"
/dev/loop5: TYPE="squashfs"
/dev/loop6: TYPE="squashfs"
/dev/loop7: TYPE="squashfs"
/dev/loop9: TYPE="squashfs"
/dev/loop8: TYPE="squashfs"
/dev/loop10: TYPE="squashfs"
/dev/loop13: TYPE="squashfs"
/dev/loop14: TYPE="squashfs"
/dev/loop11: TYPE="squashfs"
/dev/loop12: TYPE="squashfs"
/dev/loop16: TYPE="squashfs"
/dev/loop15: TYPE="squashfs"
/dev/loop17: TYPE="squashfs"
/dev/loop18: TYPE="squashfs"

I'm unsure what changed on your end to create the problem as nothing has changed at my end and I was hosting previously during the Beta and, I believe in the early version of the release.

[chalith]

In the logs there are some rootless docker related errors we are seeing.

Apr 02 17:04:51 evernode-host sagent[1729]: 20240402 17:04:51 [inf][sa] Apr 02 17:04:47 evernode-host dockerd-rootless.sh[1149722]: time="2024-04-02T17:04:47.956040968+13:00" level=error msg="failed to mount overlay: invalid argument" storage-driver=overlay2
Apr 02 17:04:51 evernode-host sagent[1729]: 20240402 17:04:51 [inf][sa] Apr 02 17:04:47 evernode-host dockerd-rootless.sh[1149722]: time="2024-04-02T17:04:47.956171462+13:00" level=error msg="exec: \"fuse-overlayfs\": executable file not found in $PATH" storage-driver=fu
Apr 02 17:04:51 evernode-host sagent[1729]: 20240402 17:04:51 [inf][sa] se-overlayfs

And

Apr 02 17:05:04 evernode-host sagent[1150315]: failed to register layer: exit status 1: "/sbin/zfs create -o mountpoint=legacy rpool/USERDATA/sashi1712030684598801003_z7vmtr/547cbf1642be80f379dfa609d47588fcd4c109f3de4896278d8464ed5e3cd05f" => cannot create 'rpool/USERDATA/>

We are trying to map the relationship between ZFS, and rootless docker storage driver.

Btw to gather more background, in the beta phase did you use the same VPS and do you remember whether the instance creation succeeded?

[rippleitinnz]

In the logs there are some rootless docker related errors we are seeing.

Apr 02 17:04:51 evernode-host sagent[1729]: 20240402 17:04:51 [inf][sa] Apr 02 17:04:47 evernode-host dockerd-rootless.sh[1149722]: time="2024-04-02T17:04:47.956040968+13:00" level=error msg="failed to mount overlay: invalid argument" storage-driver=overlay2

Apr 02 17:04:51 evernode-host sagent[1729]: 20240402 17:04:51 [inf][sa] Apr 02 17:04:47 evernode-host dockerd-rootless.sh[1149722]: time="2024-04-02T17:04:47.956171462+13:00" level=error msg="exec: \"fuse-overlayfs\": executable file not found in $PATH" storage-driver=fu

Apr 02 17:04:51 evernode-host sagent[1729]: 20240402 17:04:51 [inf][sa] se-overlayfs

And

Apr 02 17:05:04 evernode-host sagent[1150315]: failed to register layer: exit status 1: "/sbin/zfs create -o mountpoint=legacy rpool/USERDATA/sashi1712030684598801003_z7vmtr/547cbf1642be80f379dfa609d47588fcd4c109f3de4896278d8464ed5e3cd05f" => cannot create 'rpool/USERDATA/>

We are trying to map the relationship between ZFS, and rootless docker storage driver.

Btw to gather more background, in the beta phase did you use the same VPS and do you remember whether the instance creation succeeded?

• [ ]

@chalith I've never used a VPS - I run the server at my own location and it is using the same Ubuntu installation, on the same server, that I used with the Beta - in the Beta, and from memory (I'll check later) in the first live version, I had successful instance creations on multiple occasions.

I had to reinstall Evernode after a failed update in the Beta but had no errors in that and had successful instance creations following.

Also:

https://stackoverflow.com/questions/71370914/right-way-to-use-docker-rootless-mode-on-zfs-filesystem

[chalith]

Thank you @rippleitinnz In the middle of beta to current version there was an upgrade in the rootless docker installation since the way we were installing rootless docker in beta was deprecated. We are suspecting there could be an upgrade in rootless docker itself which is behind this issue. We'll do more digging and come back to you.

[rippleitinnz]

Thank you @rippleitinnz In the middle of beta to current version there was an upgrade in the rootless docker installation since the way we were installing rootless docker in beta was deprecated. We are suspecting there could be an upgrade in rootless docker itself which is behind this issue. We'll do more digging and come back to you.

OK thank you.

FYI

root@evernode-host:/home/chris# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.6 LTS
Release:        20.04
Codename:       focal

root@evernode-host:/home/chris# zfs --version
zfs-0.8.3-1ubuntu12.16
zfs-kmod-2.1.5-1ubuntu6~22.04.3

root@evernode-host:/home/chris# uname -r
5.15.0-101-generic

[kithminisg]

Hi, we've been investigating this issue, and we have some suspicions about the current rootless Docker setup, especially since it was updated in the later part of the public beta. To assist with the investigation, could you please modify the following file content as we instructed and retry to acquire an instance?

NOTE: Before making any changes, please ensure to back up the relevant file and your host account secret.

File Path: /usr/bin/sashimono/user-install.sh Line No: 190 Change: Add the attribute storage-drive as shown in the image

{ 
    "storage-driver": "overlay2"
}

[rippleitinnz]

Done @kithminisg , if you want to try to acquire an instance again.

[kithminisg]

@rippleitinnz , Have you tried to acquire an instance of this host?

[rippleitinnz]

@rippleitinnz , Have you tried to acquire an instance of this host?

I am the host, the problem was when others were trying to acquire my hosting. As I'm running on bare metal internally on my network I need someone externally to test to see if they can acquire.

host address: rnpkkEEMDSYAg1G6eHWF66kKjrKoAqMbtV https://evernode.onledger.net

As you previously did here...

Thank you. We've tried acquiring again. It seems the same error. Can you send us the logs again

[kithminisg]

@rippleitinnz , I'm sorry for assuming you were also trying to acquire, which is why I asked. I tried an acquisition, but it failed for similar reason. Could you please share the logs with us again? Thank you.

[rippleitinnz]

@rippleitinnz , I'm sorry for assuming you were also trying to acquire, which is why I asked. I tried an acquisition, but it failed for similar reason. Could you please share the logs with us again? Thank you.

Apr 19 16:10:36 evernode-host sagent[1729]: 20240419 16:10:36 [inf][sa] Apr 19 16:10:32 evernode-host dockerd-rootless.sh[1546771]: time="2024-04-19T16:10:32.959841113+12:00" level=error msg="failed to mount overlay: invalid argument" storage-driver=overlay2

Two other things. I didn't restart which I have now, just in case that was the latest issue after I made the changes. Of note: e4fa760cf68c is pulled, waited, downloaded, but never verified

I have downloaded the logs: https://pastebin.com/4cKJV8Jf

The first set is evernode.log, the second stanza is the journal:

Apr 19 16:10:41 evernode-host sagent[1729]: 20240419 16:10:41 [inf][sa] eaead16dc43b: Pulling fs layer
Apr 19 16:10:41 evernode-host sagent[1729]: 20240419 16:10:41 [inf][sa] e4fa760cf68c: Pulling fs layer
Apr 19 16:10:41 evernode-host sagent[1729]: 20240419 16:10:41 [inf][sa] b003f9770aba: Pulling fs layer
Apr 19 16:10:41 evernode-host sagent[1729]: 20240419 16:10:41 [inf][sa] ec5dfcb40d3b: Pulling fs layer
Apr 19 16:10:41 evernode-host sagent[1729]: 20240419 16:10:41 [inf][sa] 9ef0fac54d94: Pulling fs layer
Apr 19 16:10:41 evernode-host sagent[1729]: 20240419 16:10:41 [inf][sa] bfdd1585640c: Pulling fs layer
Apr 19 16:10:41 evernode-host sagent[1729]: 20240419 16:10:41 [inf][sa] e4fa760cf68c: Waiting
Apr 19 16:10:41 evernode-host sagent[1729]: 20240419 16:10:41 [inf][sa] 9ef0fac54d94: Waiting
Apr 19 16:10:41 evernode-host sagent[1729]: 20240419 16:10:41 [inf][sa] b003f9770aba: Waiting
Apr 19 16:10:41 evernode-host sagent[1729]: 20240419 16:10:41 [inf][sa] ec5dfcb40d3b: Waiting
Apr 19 16:10:43 evernode-host sagent[1729]: 20240419 16:10:43 [inf][sa] eaead16dc43b: Verifying Checksum
Apr 19 16:10:43 evernode-host sagent[1729]: 20240419 16:10:43 [inf][sa] eaead16dc43b: Download complete
Apr 19 16:10:43 evernode-host sagent[1729]: 20240419 16:10:43 [inf][sa] e4fa760cf68c: Download complete
Apr 19 16:10:45 evernode-host sagent[1729]: 20240419 16:10:45 [inf][sa] b003f9770aba: Verifying Checksum
Apr 19 16:10:45 evernode-host sagent[1729]: 20240419 16:10:45 [inf][sa] b003f9770aba: Download complete
Apr 19 16:10:46 evernode-host sagent[1729]: 20240419 16:10:46 [inf][sa] ec5dfcb40d3b: Verifying Checksum
Apr 19 16:10:46 evernode-host sagent[1729]: 20240419 16:10:46 [inf][sa] ec5dfcb40d3b: Download complete
Apr 19 16:10:48 evernode-host sagent[1729]: 20240419 16:10:48 [inf][sa] 9ef0fac54d94: Verifying Checksum
Apr 19 16:10:48 evernode-host sagent[1729]: 20240419 16:10:48 [inf][sa] 9ef0fac54d94: Download complete
Apr 19 16:10:49 evernode-host sagent[1729]: 20240419 16:10:49 [inf][sa] bfdd1585640c: Verifying Checksum
Apr 19 16:10:49 evernode-host sagent[1729]: 20240419 16:10:49 [inf][sa] bfdd1585640c: Download complete

[rippleitinnz]

@chalith @kithminisg @RichardAH @ScottChamberlain

This has never been resolved.

Since we have now moved on to v0.8.3, and it errored when I updated, I transferred my account and reinstalled Evernode.

I installed reputationd and opted in, but it errors with the acquiring of the lease, just as it did in the past.

Here is a screenshot of the transactions, which clearly shows it isn't working, and the evernode log. As this is costing fees with each error, I have now opted out of the reputationd until this gets fixed.

https://pastebin.com/ajqGLvrp

[kithminisg]

@rippleitinnz, Your VM is currently running on ZFS file system, which is not supported by rootless Docker for the moment. To work around this, you might consider maintaining a separate ext4 partition with Overlayfs support, though this is not a generic approach for us to consider.

While newer ZFS versions include overlayfs2 support, this isn't available in the Linux kernel version used by Ubuntu 20.04. We are planning an upgrade to a newer Ubuntu version, which may help to resolve this concern. In the meantime, if possible, switching to ext4 could be a better solution.

Thank you for your support and understanding.

[rippleitinnz]

@rippleitinnz, Your VM is currently running on ZFS file system, which is not supported by rootless Docker for the moment. To work around this, you might consider maintaining a separate ext4 partition with Overlayfs support, though this is not a generic approach for us to consider.

While newer ZFS versions include overlayfs2 support, this isn't available in the Linux kernel version used by Ubuntu 20.04. We are planning an upgrade to a newer Ubuntu version, which may help to resolve this concern. In the meantime, if possible, switching to ext4 could be a better solution.

Thank you for your support and understanding.

Thanks for the confirmation.

In the short term I will move to an Ext4 partition on my server, which will require me to transfer my host. I don't have a problem doing that but wonder if there is a separate transfer procedure to transfer the account created for the reputationd.

Perhaps you could give an updated procedure for transferring the Evernode instance to another server/reinstall.

[kithminisg]

@rippleitinnz,

In the next release, we'll provide a way to specify an existing reputation account that has been used previously. For the moment, you can manually set up the key file in the default reputation secret file path and try evernode reputationd opt-in if you are using the same accounts. Also, make sure the ownership and permissions for the directory and the file are set correctly.

1. Back up your both secret files in the current installation (host registration account and the reputation account) and perform the transfer operation.

2. Try re-installation. Don't opt-in to the reputation service within the installation.

3. After the successful re-installation, create the .evernode-host directory under /home/sashireputationd:

   mkdir  /home/sashireputationd/.evernode-host

4. Place the secret file of the existing reputation account with its default naming and set the permission as follows:

   chmod 440 /home/sashireputationd/.evernode-host/.host-reputationd-secret.key

5. Set the permissions and ownership for the parent directory as follows:

   chmod 550 /home/sashireputationd/.evernode-host  
   chown -R sashireputationd:sashiadmin /home/sashireputationd/.evernode-host

   6. Then try evernode reputationd opt-in.

Thank you very much. Apologies for any inconvenience.

[rippleitinnz]

Thank you for the detailed instruction. I'll backup, reformat. I'll then install Ext4 file system and Evernode and will let you know how it all goes.

[rippleitinnz]

@kithminisg

As an aside to this. To successfully transfer your Evernode if you have previously opted in to reputationd, AND the secret exists then the transfer errors out using both the transfer method from within Evernode, or using the "curl -fsSL https://raw.githubusercontent.com" method with the following error.

Do you want to set the current host account as the transferee's account? [Y/n] y

Transferring host...
Secret config file does not exist at /home/sashireputationd/.evernode-host/.host-reputationd-secret.key
MB_CLI_EXITED
Evernode transfer initiation was failed. Still do you want to continue the unistallation? [Y/n] n

It appears that if the secret exists and the conf doesn't it will error.

I opted back in then transferred successfully.

[rippleitinnz]

Closing this issue as it is now resolved.

I have reinstalled Evernode on a Ext4 partition and transferred my host successfully. ReputationD is now working as I am able to lease from my host