configure-ipfs.sh in kubernetes instructions has errors

[jasonklein]

I've deployed an IPFS cluster on k8s following the instructions in the documentation. I discovered recently that the nodes in the cluster did not have all of the configurations that should have been set by the configure-ipfs.sh script in the configmap.

The script includes setting addresses:

    ipfs config --json Addresses.API /ip4/0.0.0.0/tcp/5001
    ipfs config --json Addresses.Gateway /ip4/0.0.0.0/tcp/8080

It looks like this is the source of the issue. In the normal course, those commands throw unmarshalling errors due to the json flag. However, as part of this script, it seems to cause something peculiar to happen with the config that also triggers the earlier conditionals.

I tried removing those lines as well as using them without the json flag, but in either case the result was that the ipfs daemon was denied permission to read the config.

My hacky solution has been to remove the json flags but to include a such line with the flag after the real configurations, which seems to both allow all of the configurations to be set and to correctly manage the permissions:

# configmap.yaml
# ...
configure-ipfs.sh: |
     #!/bin/sh
    set -e
    set -x
    user=ipfs
    # This is a custom entrypoint for k8s designed to run ipfs nodes in an appropriate
    # setup for production scenarios.
    mkdir -p /data/ipfs && chown -R ipfs /data/ipfs
    if [ -f /data/ipfs/config ]; then
      if [ -f /data/ipfs/repo.lock ]; then
        rm /data/ipfs/repo.lock
      fi
      exit 0
    fi
    ipfs init --profile=badgerds,server
    ipfs config Addresses.API /ip4/0.0.0.0/tcp/5001
    ipfs config Addresses.Gateway /ip4/0.0.0.0/tcp/8080
    ipfs config --json Swarm.ConnMgr.HighWater 2000
    ipfs config --json Datastore.BloomFilterSize 1048576
    ipfs config Datastore.StorageMax 100GB

    # The line below is a hack
    ipfs config --json Addresses.API /ip4/0.0.0.0/tcp/5001

Please let me know if I've set something incorrectly or if there is an error in the configure script and/or a better solution than the one I've found.

[hsanjuan]

Hello, apparently it's missing some quotes:

ipfs config --json Addresses.API '"/ip4/0.0.0.0/tcp/5001"'
ipfs config --json Addresses.Gateway '"/ip4/0.0.0.0/tcp/8080"'

works for me (also without the --json).

but in either case the result was that the ipfs daemon was denied permission to read the config.

ok, that seems a different issue. I guess the chown call is also failing? Which ipfs docker version is it using? Who owns /data/ipfs/config , or what is the error?

to include a such line with the flag after the real configurations

Wait, what do you mean? It the excerpt you paste, the hack line a) should not work (because it uses --json) b) it's already present a couple of lines above (without --json), so I don't understand why you say this works...

[jasonklein]

Thanks for the swift reply and for updating the docs (and apologies for my own slow reply)!

I had tried putting the addresses in quotes, but I did not try double and single quotes together.

As far as the unexpected behavior I tried to explain before, I’m sorry—my explanation was confused because I still find what happened confusing. I’ll try to be clearer.

After I learned that certain configurations were not set as expected—e.g., Datastore.StorageMax showed “10GB” instead of “100GB”—I tried checking the files, given the conditional logic in the script. I added ls /data/ipfs to the top of the script, deleted and redeployed the k8s setup (including the persistent volume claims), and then checked the logs for the configure-ipfs container:

$ kubectl logs ipfs-cluster-0 configure-ipfs
badgerds
config
datastore_spec
keystore
lost+found
version
+ user=ipfs
+ mkdir -p /data/ipfs
+ chown -R ipfs /data/ipfs
+ '[' -f /data/ipfs/config ]
+ '[' -f /data/ipfs/repo.lock ]
+ exit 0

It shows that at the start of the script, ipfs has already been initialized, and, as a result, the script exits per the conditional logic because the config and lock files are present. 

So, I removed the init and config commands from the script and deleted/redeployed again:

$ kubectl logs ipfs-cluster-0 configure-ipfs
lost+found
+ user=ipfs
+ mkdir -p /data/ipfs
+ chown -R ipfs /data/ipfs
+ '[' -f /data/ipfs/config ]

This time, at the beginning of the script, the files listed are as expected—just what’s there after mounting the directory—and the script only executes the first statement of the conditional. It seemed that something about the commands was causing the init files to show up before they were expected by the script.

After comparing the config file for the original setup (with the script as stated in the docs at the time) against a basic ipfs config, I saw that the original setup’s config reflected the configurations for the badgerds and server profiles, which meant that the init command was getting executed, even though the command was not showing as executed in the configure-ipfs container’s log. The config commands after the init command, however, were not getting executed.

After some trial and error, I learned about the issue with setting the --json flag for the address configurations. When I corrected that, the configure-ipfs container’s log showed:

$ kubectl logs ipfs-cluster-0 configure-ipfs
lost+found
+ user=ipfs
+ mkdir -p /data/ipfs
+ chown -R ipfs /data/ipfs
+ '[' -f /data/ipfs/config ]
+ ipfs init '--profile=badgerds,server'
initializing IPFS node at /data/ipfs
generating 2048-bit RSA keypair...done
peer identity: Qmac4z24sh5P24onEvxMLfwoq3aP9fWTRH8i6Yqkbpmj81
to get started, enter:

        ipfs cat /ipfs/QmS4ustL54uo8FzR9455qaxZwuMiUhyvMcX9Ba8nUH4uVv/readme

+ ipfs config Addresses.API /ip4/0.0.0.0/tcp/5001
+ ipfs config Addresses.Gateway /ip4/0.0.0.0/tcp/8080
+ ipfs config --json Swarm.ConnMgr.HighWater 2000
+ ipfs config --json Datastore.BloomFilterSize 1048576
+ ipfs config Datastore.StorageMax 100GB

So, after that change, the log showed:

• ✅ the directory at the start of the script had no initialized files
• ✅ the script left the conditional after the first statement
• ✅ the script executed all of the init and config commands
• ✅ the script showed the output from calling init

Additionally, ipfs config show showed expected configurations. E.g., Datastore.StorageMax was set to “100GB” and not “10GB” ✅.

So everything seemed good.

Except then the ipfs container showed trouble:

$ kubectl logs ipfs-cluster-0 ipfs
Changing user to ipfs
ipfs version 0.4.23
Found IPFS fs-repo at /data/ipfs
Initializing daemon...
go-ipfs version: 0.4.23-6ce9a35
Repo version: 7
System version: amd64/linux
Golang version: go1.12.16

Error: open /data/ipfs/config: permission denied

More trial and error and I discovered that adding the address config command with the incorrect --json flag at the end of the script meant that:

• everything above that line was configured as intended
• the commands don’t all show in the log despite set -x
• the script still (after the init and config commands?) goes through the conditional logic and exits
• the ipfs daemon is able to read the config later

So, now the config shows the intended settings and everything appears to be running smoothly.  But I don’t understand what was wrong 😕 or why this seems to resolve the problem  🤯.

[hsanjuan]

I think the ipfs init command is not running with the ipfs user (when it runs)... therefore the configuration is getting created and edited correctly, but probably with root. When the actual container runs and tries to read it, it fails.

Can you try adding:

if [ `id -u` -eq 0 ]; then
  echo "Changing user to $user"
  # ensure folder is writable
  su-exec "$user" test -w "$repo" || chown -R -- "$user" "$repo"
  # restart script with new privileges
  exec su-exec "$user" "$0" "$@"
fi

right before the if [ -f /data/ipfs/config ]; then part ?

Probably the ipfs container ( I guess you are using a newer version) has changed and now it does not run the entrypoints directly as ipfs user.

[jasonklein]

Thanks, this worked! Except I needed to make one change:

exec su-exec "$user" sh "$0" "$@"

(Apologies, again, for the very slow reply 😬 )