search

NethermindEth / nethermind

A robust execution client for Ethereum node operators.
https://nethermind.io/nethermind-client
GNU General Public License v3.0
1.25k stars 431 forks source link

Docker non-root user fails to generate node key #3510

Closed skylenet closed 3 years ago

skylenet commented 3 years ago

Describe the bug

When running Nethermind as a docker container with a non-root user and mounting a volume that this user has access to, it fails to initialize and generate a key for the node. For whatever reason it seems to be trying to access /nethermind/Nethermind even when providing the --datadir flag.

Logs:

2021-10-11 13-47-14.1385|Configuration initialized.                                                                                                                                        
2021-10-11 13-47-14.1409|Setting BaseDbPath to: /data/nethermind_db/mainnet, from: nethermind_db/mainnet                                                                                   
2021-10-11 13-47-14.1409|Setting KeyStoreDirectory to: /data/keystore, from: keystore                                                                                                      
2021-10-11 13-47-14.1410|Setting LogDirectory to: /data/logs, from: logs
...
2021-10-11 13:37:17.2718|Generating private key for the node (no node key in configuration) - stored in plain + key store for JSON RPC unlocking                                           
2021-10-11 13:37:19.0405|Store this password for unlocking the node key for JSON RPC - this is not secure - this log message will be in your log files. Use only in DEV contexts.          
2021-10-11 13:37:19.0405|aeXUCyFi
2021-10-11 13:37:19.0703|Step SetupKeyStore            failed after 1881ms System.Security.Cryptography.CryptographicException: An error occurred while trying to encrypt the provided data
. Refer to the inner exception for more information.                                                                                                                                       
 ---> System.UnauthorizedAccessException: Access to the path '/nethermind/Nethermind' is denied.                                                                                           
 ---> System.IO.IOException: Permission denied                                                                                                                                             
   --- End of inner exception stack trace ---                                                                                                                                              
   at System.IO.FileSystem.CreateDirectory(String fullPath)                                                                                                                                
   at System.IO.DirectoryInfo.Create()                                                                                                                                                     
   at Microsoft.AspNetCore.DataProtection.Repositories.FileSystemXmlRepository.GetAllElementsCore()+MoveNext()                                                                             
   at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)                                                                                                                    
   at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)                                                                                                                         
   at Microsoft.AspNetCore.DataProtection.Repositories.FileSystemXmlRepository.GetAllElements()                                                                                            
   at Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager.GetAllKeys()                                                                                                         
   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingProvider.CreateCacheableKeyRingCore(DateTimeOffset now, IKey keyJustAdded)                                                  
   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingProvider.Microsoft.AspNetCore.DataProtection.KeyManagement.Internal.ICacheableKeyRingProvider.GetCacheableKeyRing(DateTimeOf
fset now)                                                                                                                                                                                  
   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingProvider.GetCurrentKeyRingCore(DateTime utcNow, Boolean forceRefresh)                                                       
   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingProvider.GetCurrentKeyRing()                                                                                                
   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.Protect(Byte[] plaintext)                                                                                
   --- End of inner exception stack trace ---                                                                                                                                              
   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.Protect(Byte[] plaintext)                                                                                
   at Nethermind.Crypto.ProtectedData.AspNetWrapper.Protect(Byte[] userData, Byte[] optionalEntropy, DataProtectionScope scope) in /src/Nethermind/Nethermind.Crypto/ProtectedData.AspNetWr
apper.cs:line 34                                                                                                                                                                           
   at Nethermind.Crypto.ProtectedData.Protect(Byte[] userData, Byte[] optionalEntropy, DataProtectionScope scope) in /src/Nethermind/Nethermind.Crypto/ProtectedData.cs:line 39            
   at Nethermind.Crypto.ProtectedData`1.Protect(Byte[] data) in /src/Nethermind/Nethermind.Crypto/ProtectedData.T.cs:line 46                                                               
   at Nethermind.Crypto.ProtectedData`1..ctor(Byte[] data, ICryptoRandom random, ITimestamper timestamper) in /src/Nethermind/Nethermind.Crypto/ProtectedData.T.cs:line 40                 
   at Nethermind.Crypto.ProtectedPrivateKey..ctor(PrivateKey privateKey, ICryptoRandom random, ITimestamper timestamper) in /src/Nethermind/Nethermind.Crypto/ProtectedPrivateKey.cs:line 2
5                                                                                                                                                                                          
   at Nethermind.Wallet.NodeKeyManager.<LoadNodeKey>g__LoadKeyFromFile|9_0() in /src/Nethermind/Nethermind.Wallet/NodeKeyManager.cs:line 88                                                
   at Nethermind.Wallet.NodeKeyManager.LoadNodeKey() in /src/Nethermind/Nethermind.Wallet/NodeKeyManager.cs:line 110                                                                       
   at Nethermind.Init.Steps.SetupKeyStore.<>c__DisplayClass2_0.<Execute>b__0() in /src/Nethermind/Nethermind.Init/Steps/SetupKeyStore.cs:line 73                                           
   at System.Threading.Tasks.Task.InnerInvoke()                                                                                                                                            
   at System.Threading.Tasks.Task.<>c.<.cctor>b__277_0(Object obj)                                                                                                                         
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)                  
--- End of stack trace from previous location ---                                                                                                                                          
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)                  
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)                                                                                   
--- End of stack trace from previous location ---                                                                                                                                          
   at Nethermind.Init.Steps.SetupKeyStore.Execute(CancellationToken cancellationToken) in /src/Nethermind/Nethermind.Init/Steps/SetupKeyStore.cs:line 44

To Reproduce Steps to reproduce the behavior:

  1. Run the following command on your local machine (use a non root user):
mkdir -p data && docker run --rm -it -v $PWD/data:/data -u $UID nethermind/nethermind:latest --datadir=/data
  1. Check the logs

Expected behavior A new node key to be created for the node within the /data/keystore directory (--KeyStore.KeyStoreDirectory is by default "keystore")

AntiD2ta commented 3 years ago

@skylenet This is an issue we are faced before when we wanted to create a rootless docker image. When you bind a folder in your host with the volume folder inside the container, docker sets as uid:gid of that folder root:root even if we chown that folder inside the Dockerfile. Later we find out this is an open issue of docker. There are a lot of proposed workarounds but none of them worked for us.

skylenet commented 3 years ago

@skylenet This is an issue we are faced before when we wanted to create a rootless docker image. When you bind a folder in your host with the volume folder inside the container, docker sets as uid:gid of that folder root:root even if we chown that folder inside the Dockerfile. Later we find out this is an open issue of docker. There are a lot of proposed workarounds but none of them worked for us.

Hey @AntiD2ta . Thanks for looking into it and pointing me to that docker issue. I've looked into it and I'm not sure if it's really related to that. In my case the volume is mounted with the correct UID and the Nethermind process does have access to write into it.

After having a closer look into the stack trace and code, this seems to be happening around here: https://github.com/NethermindEth/nethermind/blob/1d2f21ba04d4fd14cada95479d57b8d368b70af7/src/Nethermind/Nethermind.Crypto/ProtectedData.AspNetWrapper.cs#L68

From what it seems like, it will try to create the /nethermind/Nethermind directory for the DataProtectionProvider. I think this might be a bug and that this should also be somewhere within the keystore directory.

As a temporary workaround for docker users, you can additionally create a volume mount for /nethermind/Nethermind and then the key generation will work. Example:

# Create the directories with your user so that they will also have the right permissions when mounted
mkdir data appdata  
# Mount data and appdata dir
docker run --rm -it -v $PWD/data:/data -v $PWD/appdata:/nethermind/Nethermind -u $UID nethermind/nethermind:latest --datadir=/data

Update:

Another workaround can also be to change the workdir and the entrypoint so that the "Nethermind" directory will be created within a volume mount that has permissions . For example:

mkdir data
docker run --rm -it -v $PWD/data:/data --workdir=/data --entrypoint=/nethermind/Nethermind.Runner -u $UID nethermind/nethermind:latest --datadir=/data
AntiD2ta commented 3 years ago

@skylenet Thanks very much for the feedback, we will again look into it and try to build a rootless docker image so it will be easier for you to run a container as non-root.

attila-lendvai commented 1 year ago

EDIT: i have found the right open issue for the error i have reported below:

https://github.com/NethermindEth/nethermind/issues/4305

i'm also seeing this:

System.UnauthorizedAccessException: Access to the path '/Nethermind' is denied.

when starting it as:

/gnu/store/24zb6izdz5l0l6cd65fb8mmvlwb6gmwm-nethermind-binary-1.14.7/bin/Nethermind.Runner --Sync.UseGethLimitsInFastBlocks=true --Sync.FastSync=true --HealthChecks.UIEnabled=true --HealthChecks.Enabled=true --JsonRpc.JwtSecretFile=/var/lib/nethermind/gnosis/jwt-secret --JsonRpc.EnginePort=8551 --datadir=/var/lib/nethermind/gnosis --config=xdai

context: working on a Guix service to start Nethermind. it's started as non-root, under a custom user and group.

@AntiD2ta shall i report this as a new issue? or is this a user error?

full log:

2022-12-19 00:56:24 2022-12-19 00-56-09.5317|Nethermind starting initialization.
2022-12-19 00:56:24 2022-12-19 00-56-09.7028|Loading embedded plugins
2022-12-19 00:56:24 2022-12-19 00-56-09.7029|  Found plugin type Nethermind.Consensus.AuRa.AuRaPlugin
2022-12-19 00:56:24 2022-12-19 00-56-09.7029|  Found plugin type Nethermind.Consensus.Clique.CliquePlugin
2022-12-19 00:56:24 2022-12-19 00-56-09.7030|  Found plugin type Nethermind.Consensus.Ethash.EthashPlugin
2022-12-19 00:56:24 2022-12-19 00-56-09.7030|  Found plugin type Nethermind.Consensus.Ethash.NethDevPlugin
2022-12-19 00:56:24 2022-12-19 00-56-09.7030|  Found plugin type Nethermind.Hive.HivePlugin
2022-12-19 00:56:24 2022-12-19 00-56-09.7030|  Found plugin type Nethermind.UPnP.Plugin.UPnPPlugin
2022-12-19 00:56:24 Resolved executing directory as /gnu/store/24zb6izdz5l0l6cd65fb8mmvlwb6gmwm-nethermind-binary-1.14.7/share/nethermind-binary-1.14.
7.
2022-12-19 00:56:24 2022-12-19 00-56-09.7553|Loading 9 assemblies from /gnu/store/24zb6izdz5l0l6cd65fb8mmvlwb6gmwm-nethermind-binary-1.14.7/share/neth
ermind-binary-1.14.7/plugins
2022-12-19 00:56:24 2022-12-19 00-56-09.7555|Loading assembly Nethermind.Merge.AuRa
2022-12-19 00:56:24 2022-12-19 00-56-09.7601|  Found plugin type Nethermind.Merge.AuRa
2022-12-19 00:56:24 2022-12-19 00-56-09.7601|Loading assembly Nethermind.Merge.Plugin
2022-12-19 00:56:24 2022-12-19 00-56-09.7690|  Found plugin type Nethermind.Merge.Plugin
2022-12-19 00:56:24 2022-12-19 00-56-09.7690|Loading assembly Nethermind.HealthChecks
2022-12-19 00:56:24 2022-12-19 00-56-09.7774|  Found plugin type Nethermind.HealthChecks
2022-12-19 00:56:24 2022-12-19 00-56-09.7774|Loading assembly Nethermind.AccountAbstraction
2022-12-19 00:56:24 2022-12-19 00-56-09.7828|  Found plugin type Nethermind.AccountAbstraction
2022-12-19 00:56:24 2022-12-19 00-56-09.7828|Loading assembly Nethermind.Mev
2022-12-19 00:56:24 2022-12-19 00-56-09.7909|  Found plugin type Nethermind.Mev
2022-12-19 00:56:24 2022-12-19 00-56-09.7909|Loading assembly Nethermind.Init
2022-12-19 00:56:24 2022-12-19 00-56-09.7933|Loading assembly Nethermind.Consensus.AuRa
2022-12-19 00:56:24 2022-12-19 00-56-09.8001|Loading assembly Nethermind.Api
2022-12-19 00:56:24 2022-12-19 00-56-09.8053|Loading assembly Nethermind.EthStats
2022-12-19 00:56:24 2022-12-19 00-56-09.8074|  Found plugin type Nethermind.EthStats
2022-12-19 00:56:24 2022-12-19 00-56-09.9664|Loading standard NLog.config file from /gnu/store/24zb6izdz5l0l6cd65fb8mmvlwb6gmwm-nethermind-binary-1.14
.7/share/nethermind-binary-1.14.7/NLog.config.
2022-12-19 00:56:24 2022-12-19 00-56-10.8489|NLog.config loaded in 882ms.
2022-12-19 00:56:24 2022-12-19 00-56-10.8574|Reading config file from /gnu/store/24zb6izdz5l0l6cd65fb8mmvlwb6gmwm-nethermind-binary-1.14.7/share/nethe
rmind-binary-1.14.7/configs/xdai.cfg
2022-12-19 00:56:24 2022-12-19 00-56-11.6494|Configuration initialized.
2022-12-19 00:56:24 2022-12-19 00-56-11.6607|Setting BaseDbPath to: /var/lib/nethermind/gnosis/nethermind_db/xdai, from: nethermind_db/xdai
2022-12-19 00:56:24 2022-12-19 00-56-11.6608|Setting KeyStoreDirectory to: /var/lib/nethermind/gnosis/keystore, from: keystore
2022-12-19 00:56:24 2022-12-19 00-56-11.6608|Setting LogDirectory to: /var/lib/nethermind/gnosis/logs, from: logs
2022-12-19 00:56:24 2022-12-19 00:56:11.6960|RocksDb Version: 6.29.3 
2022-12-19 00:56:24 2022-12-19 00:56:11.7353|Loading chainspec from embedded resources: /gnu/store/24zb6izdz5l0l6cd65fb8mmvlwb6gmwm-nethermind-binary-
1.14.7/share/nethermind-binary-1.14.7/chainspec/xdai.json 
2022-12-19 00:56:24 2022-12-19 00:56:12.4086|Using http://ipv4.icanhazip.com to get external ip 
2022-12-19 00:56:24 2022-12-19 00:56:12.8262|Setting up memory allowances 
2022-12-19 00:56:24 2022-12-19 00:56:12.8262|  memory hint:        768MB 
2022-12-19 00:56:24 2022-12-19 00:56:12.8262|  general memory:     32MB 
2022-12-19 00:56:24 2022-12-19 00:56:12.8262|  peers memory:       50MB 
2022-12-19 00:56:24 2022-12-19 00:56:12.8297|  Netty memory:       33MB 
2022-12-19 00:56:24 2022-12-19 00:56:12.8297|  mempool memory:     114MB 
2022-12-19 00:56:24 2022-12-19 00:56:12.8297|  fast blocks memory: 53MB 
2022-12-19 00:56:24 2022-12-19 00:56:12.8297|  trie memory:        96MB 
2022-12-19 00:56:24 2022-12-19 00:56:12.8454|  DB memory:          387MB 
2022-12-19 00:56:24 2022-12-19 00:56:13.1844|Generating private key for the node (no node key in configuration) - stored in plain + key store for JSON
 RPC unlocking 
2022-12-19 00:56:24 2022-12-19 00:56:17.9453|Store this password for unlocking the node key for JSON RPC - this is not secure - this log message will 
be in your log files. Use only in DEV contexts. 
2022-12-19 00:56:24 2022-12-19 00:56:18.0093|Step SetupKeyStore            failed after 4837ms System.Security.Cryptography.CryptographicException: An
 error occurred while trying to encrypt the provided data. Refer to the inner exception for more information. For more information go to http://aka.ms
/dataprotectionwarning
2022-12-19 00:56:24  ---> System.UnauthorizedAccessException: Access to the path '/Nethermind' is denied.
2022-12-19 00:56:24  ---> System.IO.IOException: Permission denied
2022-12-19 00:56:24    --- End of inner exception stack trace ---
2022-12-19 00:56:24    at System.IO.FileSystem.CreateDirectory(String fullPath)
2022-12-19 00:56:24    at System.IO.DirectoryInfo.Create()
2022-12-19 00:56:24    at Microsoft.AspNetCore.DataProtection.Repositories.FileSystemXmlRepository.GetAllElementsCore()+MoveNext()
2022-12-19 00:56:24    at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
2022-12-19 00:56:24    at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
2022-12-19 00:56:24    at Microsoft.AspNetCore.DataProtection.Repositories.FileSystemXmlRepository.GetAllElements()
2022-12-19 00:56:24    at Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager.GetAllKeys()
2022-12-19 00:56:24    at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingProvider.CreateCacheableKeyRingCore(DateTimeOffset now, IKey keyJus
tAdded)
2022-12-19 00:56:24    at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingProvider.Microsoft.AspNetCore.DataProtection.KeyManagement.Internal
.ICacheableKeyRingProvider.GetCacheableKeyRing(DateTimeOffset now)
2022-12-19 00:56:24    at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingProvider.GetCurrentKeyRingCore(DateTime utcNow, Boolean forceRefres
h)
2022-12-19 00:56:24    at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingProvider.GetCurrentKeyRing()
2022-12-19 00:56:24    at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.Protect(Byte[] plaintext)
2022-12-19 00:56:24    --- End of inner exception stack trace ---
2022-12-19 00:56:24    at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.Protect(Byte[] plaintext)
2022-12-19 00:56:24    at Nethermind.Crypto.ProtectedData.AspNetWrapper.Protect(Byte[] userData, Byte[] optionalEntropy, DataProtectionScope scope)
2022-12-19 00:56:24    at Nethermind.Crypto.ProtectedData.Protect(Byte[] userData, Byte[] optionalEntropy, DataProtectionScope scope)
2022-12-19 00:56:24    at Nethermind.Crypto.ProtectedData`1.Protect(Byte[] data)
2022-12-19 00:56:24    at Nethermind.Crypto.ProtectedData`1..ctor(Byte[] data, ICryptoRandom random, ITimestamper timestamper)
2022-12-19 00:56:24    at Nethermind.Crypto.ProtectedPrivateKey..ctor(PrivateKey privateKey, ICryptoRandom random, ITimestamper timestamper)
2022-12-19 00:56:24    at Nethermind.Wallet.NodeKeyManager.<LoadNodeKey>g__LoadKeyFromFile|9_0()
2022-12-19 00:56:24    at Nethermind.Wallet.NodeKeyManager.LoadNodeKey()
2022-12-19 00:56:24    at Nethermind.Init.Steps.SetupKeyStore.<>c__DisplayClass2_0.<Execute>b__0()
2022-12-19 00:56:24    at System.Threading.Tasks.Task.InnerInvoke()
2022-12-19 00:56:24    at System.Threading.Tasks.Task.<>c.<.cctor>b__272_0(Object obj)
2022-12-19 00:56:24    at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, 
ContextCallback callback, Object state)
2022-12-19 00:56:24 --- End of stack trace from previous location ---
2022-12-19 00:56:24    at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, 
ContextCallback callback, Object state)
2022-12-19 00:56:24    at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
2022-12-19 00:56:24 --- End of stack trace from previous location ---
2022-12-19 00:56:24    at Nethermind.Init.Steps.SetupKeyStore.Execute(CancellationToken cancellationToken)
2022-12-19 00:56:24    at Nethermind.Init.Steps.EthereumStepsManager.ExecuteStep(IStep step, StepInfo stepInfo, CancellationToken cancellationToken)
AntiD2ta commented 1 year ago

@attila-lendvai thanks for reporting. It seems to me this is an user error. By any chance are you keen to join our discord server (general channel) and get a more dynamic and faster feedback flow? You will get all the help you need there. If you can't join our server, then please report this in a separate issue.

You can find our discord server URL in our official website: https://nethermind.io

attila-lendvai commented 1 year ago

@AntiD2ta since then i have found an open issue that seems to be about my error (with a proposed PR to fix it):

https://github.com/NethermindEth/nethermind/issues/4305

i also joined the discord channel, you can catch me there, but i think we should use that issue to discuss it.