Closed qhartman closed 4 years ago
In studying the initial init hook output more closely, I've found that there is a group of "Binlinking..." entries that have no corresponding "Binlinked..." entry in the list. They are always together, and always at the end of the list. They are the following:
active-oversight-api.default hook[init]:(HK): » Binlinking curl-config from core/curl/7.54.1/20180608142121 into /bin
active-oversight-api.default hook[init]:(HK): » Binlinking eu-nm from core/elfutils/0.170/20180608141148 into /bin
active-oversight-api.default hook[init]:(HK): » Binlinking file from core/file/5.32/20180608050620 into /bin
active-oversight-api.default hook[init]:(HK): » Binlinking fc-validate from core/fontconfig/2.11.95/20180717232730 into /bin
active-oversight-api.default hook[init]:(HK): » Binlinking xml2-config from core/libxml2/2.9.6/20180608141053 into /bin
active-oversight-api.default hook[init]:(HK): » Binlinking xsltproc from core/libxslt/1.1.31/20180608141131 into /bin
active-oversight-api.default hook[init]:(HK): » Binlinking node from core/node/10.7.0/20180723184209 into /bin
active-oversight-api.default hook[init]:(HK): » Binlinking pg_upgrade from core/postgresql/9.6.9/20180716151728 into /bin
active-oversight-api.default hook[init]:(HK): » Binlinking ruby from core/ruby/2.4.2/20180619210241 into /bin
active-oversight-api.default hook[init]:(HK): » Binlinking strace-graph from core/strace/4.23/20180725141600 into /bin
active-oversight-api.default hook[init]:(HK): » Binlinking sudoedit from core/sudo/1.8.18p1/20180609192018 into /bin
active-oversight-api.default hook[init]:(HK): » Binlinking blkid from core/util-linux/2.31.1/20180608101132 into /bin
active-oversight-api.default hook[init]:(HK): » Binlinking i386 from core/util-linux/2.31.1/20180608101132 into /bin
active-oversight-api.default hook[init]:(HK): » Binlinking montage from finalze/imagemagick/6.9.2-10/20180801164036 into /bin
That list is consistent from one init to the next. There is a corresponding number of messages like the following later in the log output:
active-oversight-api.default hook[init]:(HK): ✗✗✗
active-oversight-api.default hook[init]:(HK): ✗✗✗ Permission denied (os error 13)
active-oversight-api.default hook[init]:(HK): ✗✗✗
active-oversight-api.default hook[init]:(HK): ✗✗✗
active-oversight-api.default hook[init]:(HK): ✗✗✗ Permission denied (os error 13)
active-oversight-api.default hook[init]:(HK): ✗✗✗
active-oversight-api.default hook[init]:(HK): ✗✗✗
active-oversight-api.default hook[init]:(HK): ✗✗✗ Permission denied (os error 13)
active-oversight-api.default hook[init]:(HK): ✗✗✗
active-oversight-api.default hook[init]:(HK): ✗✗✗
active-oversight-api.default hook[init]:(HK): ✗✗✗ Permission denied (os error 13)
active-oversight-api.default hook[init]:(HK): ✗✗✗
active-oversight-api.default hook[init]:(HK): ✗✗✗
active-oversight-api.default hook[init]:(HK): ✗✗✗ Permission denied (os error 13)
active-oversight-api.default hook[init]:(HK): ✗✗✗
active-oversight-api.default hook[init]:(HK): ✗✗✗
active-oversight-api.default hook[init]:(HK): ✗✗✗ Permission denied (os error 13)
active-oversight-api.default hook[init]:(HK): ✗✗✗
active-oversight-api.default hook[init]:(HK): ✗✗✗
active-oversight-api.default hook[init]:(HK): ✗✗✗ Permission denied (os error 13)
active-oversight-api.default hook[init]:(HK): ✗✗✗
active-oversight-api.default hook[init]:(HK): ✗✗✗
active-oversight-api.default hook[init]:(HK): ✗✗✗ Permission denied (os error 13)
active-oversight-api.default hook[init]:(HK): ✗✗✗
active-oversight-api.default hook[init]:(HK): ✗✗✗
active-oversight-api.default hook[init]:(HK): ✗✗✗ Permission denied (os error 13)
active-oversight-api.default hook[init]:(HK): ✗✗✗
active-oversight-api.default hook[init]:(HK): ✗✗✗
active-oversight-api.default hook[init]:(HK): ✗✗✗ Permission denied (os error 13)
active-oversight-api.default hook[init]:(HK): ✗✗✗
active-oversight-api.default hook[init]:(HK): ✗✗✗
active-oversight-api.default hook[init]:(HK): ✗✗✗ Permission denied (os error 13)
active-oversight-api.default hook[init]:(HK): ✗✗✗
active-oversight-api.default hook[init]:(HK): ∅ Skipping binlink because blkid already exists at /bin/blkid. Use --force to overwrite
active-oversight-api.default hook[init]:(HK): ✗✗✗
active-oversight-api.default hook[init]:(HK): ✗✗✗ Permission denied (os error 13)
active-oversight-api.default hook[init]:(HK): ✗✗✗
active-oversight-api.default hook[init]:(HK): ✗✗✗
active-oversight-api.default hook[init]:(HK): ✗✗✗ Permission denied (os error 13)
active-oversight-api.default hook[init]:(HK): ✗✗✗
So, aside from the one entry for blkid, there seems to be something happening in parallel that's changing the permissions on the fs and preventing the links from being created by the init hook. This would also explain why my attempt at directly linking the node binary is failing.
We're going to dig a bit further into this to figure out exactly what's going on.
Pinging @fnichol to get his thoughts, too.
Great, thanks! Let me know if there's any additional info I can provide that will be useful. Based on what I've seen it seems that having the init hook run as root would effectively get around this problem, but I'm not sure that's desirable. It seems it may be given that init hooks are intended to be doing system-setup type stuff before the application starts, but it certainly would open up a risk vector. I'm really curious why most of them seem to work though, so I'm interested in your insights.
I'm also curious if we don't want the init hook to be running as root, if there's an accepted escape hatch for cases where you do need to do something as root (as I also mentioned here https://forums.habitat.sh/t/binlinking-says-its-linking-bins-but-isnt/744/6) because simply installing sudo and using it didn't do the trick.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. We value your input and contribution. Please leave a comment if this issue still affects you.
In general, I don't think we want to be creating arbitrary binlinks from inside a package. Accessing binaries from dependency packages should happen already by virtue of them being on the service's path, or by using pkgPathFor
explicitly in a hook.
Originally posted some detail here: https://forums.habitat.sh/t/binlinking-says-its-linking-bins-but-isnt/744
Briefly, binlinking isn't behaving consistently when I use it in my init hook. There are a handful of packages it doesn't seem to be actually creating binlinks for, even though the log output indicates it is. Notably core/node and core/strace seem to be failing.
I've also added a manual linking of the node executable to my init:
ln -s {{pkgPathFor "core/node"}}/bin/node /bin/node
which ALSO seems to be failing silently, so maybe there is a common root cause here.
If I re-run my generated init hook by hand after starting the container, the missing links are created correctly:
using hab 0.59, packaged into a docker container.
Is there any other information I could provide that would be useful?