glibc 2.38 and crypt idiocy

[terefang]

i was reviewing the changes to toybox 0.8.10 and noticed that you chose to declare crypt in case glibc is used.

actually the case is not that easy, citing glibc 2.38 news:

libcrypt is no longer built by default; one may use the "--enable-crypt" option to build libcrypt. libcrypt is likely to be removed from the GNU C Library in a future release, so it is recommended that applications port away from it to an alternative such as libxcrypt.

at the time of this writing glibc 2.38 is only experimental in Debian: https://packages.debian.org/source/experimental/glibc

at the time of this writing glibc 2.38 is blocked for Ubuntu: https://ubuntu-archive-team.ubuntu.com/proposed-migration/update_excuses.html#glibc

at the time of this writing buildroot 2023.8 is still on glibc 2.37.

the state for Fedora/Centos/RHEL going forward is documented here: https://fedoraproject.org/wiki/Changes/Replace_glibc_libcrypt_with_libxcrypt

it looks like Arch has followed the Fedora approach and migrated to libxcrypt: https://archlinux.org/packages/core/x86_64/libxcrypt/

it looks like Gentoo has followed the Fedora approach also: https://www.gentoo.org/support/news-items/2021-10-18-libxcrypt-migration-stable.html

looks like Yocto/OpenEmbedded also migrated to libxcrypt with glibc 2.38.

long story short:

if you build against glibc 2.38 or later, dont declare crypt but rather include crypt.h and link against libxcrypt with -lcrypt.

musl-libc seems fine since:

• crypt is declared in unistd.h
• has also crypt.h with crypt

hope that helps

[landley]

Can they migrate to musl?

[terefang]

i am just being helpful getting the info out in advance so that such a useful tool as toybox keeps building everywhere most of the time.

but i am fine if you decide to close with "wont fix", rather directing other source-compilers to link statically with musl.

in my case a already changed my build system to link toyboy statically to musl.

the only other problematic area i see is PAM, but that isnt supported by toybox today anyhow.

[landley]

Sorry for the sarcasm, I'm just tired of glibc breaking stuff (2.36 broke mount.h for example, although we got them to revert that one: https://github.com/landley/toybox/issues/362).

I have a pending rewrite of lib/password.c that removes the need for shadow.h (among other things so it can build on bionic) and I already have md5sum, sha256, and sha512 implementations in toybox. The one I'm missing is des, and my TODO item was to decide whether to implement it or just punt and say our default is md5sum instead. (Which isn't any less secure than 3des. The downside would be inability to log in using old des passwords...)

But yes, I intend to fix this. And given glibc 2.38 breaking stuff, it needs to bump up near the top of the todo list...

[terefang]

let me just summarize – that would mean you would support md5 ($1$...), sha256 ($5$...) and sha512 ($6$...) hashed passwords without dependency.

linking to libxcrypt might pollute your license since it is LGPL-2.1 and others.

so what you need are (preferable 0-) BSD licensed codes for des, blowfish and maybe others as implemented by libxcrypt, eg:

yescrypt       $y$
gost_yescrypt  $gy$
scrypt         $7$
bcrypt         $2b$
bcrypt_y       $2y$
bcrypt_a       $2a$
bcrypt_x       $2x$
sha512crypt    $6$
sha256crypt    $5$
sha1crypt      $sha1
sunmd5         $md5
md5crypt       $1$
nt             $3$
bsdicrypt      _
bigcrypt       :
descrypt       :

[enh-google]

aren't 1, 5, and 6 the only ones actually implemented by glibc, so the others don't matter? https://man7.org/linux/man-pages/man3/crypt.3.html --- i thought "we don't want to bother adding others" was the rationale from dropping crypt() from glibc?

[terefang]

yes but password crypto has moved on since sha512, so now yescrypt and scrypt are the new security standards and all other for downward compatiblity.

dont get me wrong, imho it was a good move to spin password crypto out of the libc into a library that can be more rapidly upgraded.

now the bad part is that various libc implemented it as part of posix compliance and glibc 2.38 breaks some userlands hard,

[landley]

It's not just the license, toybox's default configuration does not depend on any external libraries except libc (which includes libm and friends provided by the libc package). This is explicit policy ("Shared Libraries" section) in https://landley.net/toybox/design.html

And it's not just glibc providing specific algorithms, toybox itself currently only implements those 4 algorithms in its string detection: https://github.com/landley/toybox/blob/master/lib/password.c#L15

Shadow passwords were invented (in the previous century) because it doesn't matter what the algorithm is: if you have the hashes, they can crack the passwords at leisure with compute clusters, so the hashes should not be world readable. Switching algorithms just moves it from "do it on my laptop" to "do it with $50 of cloud time". And that's handwaving past the crypto bros with warehouses full of graphics cards offering hash cracking as a service sidelines...

My concern with not supporting des was compatibility with existing /etc/passwd entries (especially VMs slapped together with echo $ACCOUNT >> new/etc/passwd), but A) you're convincing me that no finite array of hashes is sufficient for that sort of thing anymore, B) md5sum was already the default in Red Hat 6.2 from y2k (and probably before that, but that's the oldest VM image I have lying around). DES was intentionally broken because of US export regulations that were nerfed back at the turn of the century.

I can add more hashing algorithms, and occasionally try to look up what $id$ is assigned to sha3 since we've already got that, but people like https://passlib.readthedocs.io/en/stable/modular_crypt_format.html instead complain that they don't like the $id$ format and insist everybody should move to their own hand-rolled thing...

But it's not high on the todo list. I have a todo item to implement blake2 because b2sum showed up in my devuan default install (apparently it's in coreutils now), but it's not a priority unless somebody shows up with an existing use case or maybe busybox adds it...

[terefang]

also the busybox alignment is a good topic

just looked up busybox and it does have des, md5 and compile-time option sha256/512

so toybox does not need to be more catholic than the pope !

i know passlib/mcf very well, have had some heated discussions about it in my dayjob.

looking at common linux distros, it seems that the current gold-standard for passwd/shadow compatible passwords is yescrypt (ie. $7$...), followed by scrypt and pbkdf, with bcrypt/sha512 being the previous one.

since toybox is written in c, i might be swayed to create some pull requests

[terefang]

let me make a quick braindump how this all relates to toybox commands/toys:

• passwd toy to set user password in passwd/shadow
• mkpasswd toy to generate crypted password for "other" files
• chsh toy verifies user password (ie uses crypt)
• login toy verifies user password (ie uses crypt)
• su toy verifies user password (ie uses crypt)
• sulogin toy verifies user password (ie uses crypt)

did i miss something ?

[terefang]

real-world use case: mkpasswd could double as htpasswd https://httpd.apache.org/docs/2.4/programs/htpasswd.html

$ htpasswd -nb -m root secret 
root:$apr1$Ld1RWqJw$.ftjqSg1JR6SWsLHOvLlo/

$ htpasswd -nb -B root secret 
root:$2y$05$2O6aUQyic5QBB0.XI7.1YO8f.kygINcgHZpZu4zouIG05s6APkboy

$ htpasswd -nb -d root secret 
root:4QDQCqFzgVzhU

$ htpasswd -nb -s root secret 
root:{SHA}5en6G6MezRroT3XKqkdPOmY/BfQ=

so we get the following algorithms:

• apr1 (ie salted md5)
• bcrypt
• des
• sha1

[terefang]

real-world use: replace caddy hash-password

$ caddy hash-password -algorithm scrypt -plaintext secret
AIWb7bWYCI9Q1w+ltqFGhe+JrVh+DIQHgu1arEZ1I48=

$ caddy hash-password -algorithm bcrypt -plaintext secret
JDJhJDE0JHkvM0lEanlOT0h0d3IwWXVoSy5TQnV5dVJKSU9TUnNaTFFkRnM3bmk3N2pTUTFoZW9IVjNx

so we get the following algorithms:

• bcrypt (is actually base64 wrapped unix/bcrypt)
• scrypt (is actually base64 wrapped pbkdf with sha1/256/512)

[terefang]

real-world use: nginx basic auth module http://nginx.org/en/docs/http/ngx_http_auth_basic_module.html

in addition to htpasswd style it also support rfc2307 used in ldap

so we get the following algorithms:

• crypt/mcf ( ie. {crypt} prefixed unix-mcf)
• md5
• sha1
• smd5 (ie salted md5)
• ssha (ie salted sha1/256/512)

[terefang]

real-world use: syslinux sha1pass https://stackoverflow.com/questions/3774815/what-does-the-4-mean-in-the-output-of-sha1pass https://github.com/TritonDataCenter/syslinux/blob/master/utils/sha1pass

$ sha1pass secret
$4$GTdnmykS$2SiwV+ruXRwor4pUmKFS7uXHj70$

so we get the following algorithms:

• salted sha1 mcf

[terefang]

real-world use: grub menu password https://help.ubuntu.com/community/Grub2/Passwords

$ grub-mkpasswd-pbkdf2 
Enter password: 
Reenter password: 
PBKDF2 hash of your password is grub.pbkdf2.sha512.10000.3099FFBE3926D7B016D73072D67E9ED40445A8D93DAD3710F673C3CB78A6145C29245B38E5D02234F2EAB11809B413B5B4019A79462246BD8FD208207656E5D7.917BD5125B076EC7A1B9FFD215601E580F1388173D9065E2C581C25598F7659DDC2C14F1C09AE146E0CD8FBBA2055A2B1D016A0C18A26E4AECD11F9B2B148768

so we get the following algorithms:

• pbkdf2/sha512

[terefang]

XSH/POSIX (IEEE Std 1003.1-2017) does only specify a DES salt for crypt https://pubs.opengroup.org/onlinepubs/9699919799/functions/crypt.html

linux man pages only note glibc feature examples md5, sha256/512 and bcrypt for non-glibc https://man7.org/linux/man-pages/man3/crypt.3.html

[landley]

The bug here is "glibc broke existing code". The fix is "toybox switches to using the hashes it already has internally".

I coded up about half of that today (laptop+coffee shop often translates to "work offline", it's less distracting), but I'm also switching the /etc/passwd reading and writing code to internal stuff that uses a common codepath for all the colon separated files and doesn't need shadow.h (which gets building defconfig against the NDK a lot closer), so I'm trying to chip that apart and do smaller checkins rather than making the big hairball bigger. (And friday is spent in airports.)

I've already said I have no interest in any of the external libraries you're going on about, or adding new hashes we haven't already got in response to a glibc regression. If I was going to add hashes out of a library, Elliott already hooked up boringssl to md5sum.c back in 2016 (commit adef5dcb1857) so the logical thing to do would be see what that's got. But a glibc bug does not obligate toybox to feature creep.

A request for a new hash is a request for a new feature. I'm not categorically against the concept (and mentioned blake2 on the post-1.0 todo list, along with screen and rsync), but "glibc got worse" is not a reason for it, and "I want many all at once" means you didn't need a specific one. (And picking one now would be kind of disingenuous: you've already established that you want more because more.)

I implemented sha3. It's still not used in /etc/passwd, and I never got a clear explanation why. I asked what /etc/passwd prefix ID sha3 should have, and you posted about grub, syslinux, nginx, and apache. Toybox doesn't implement a bootloader, and our httpd is intentionally very simple.

I'm only familiar with blake2 because over the pandemic I helped out a friend with a project that collapsed into crypto mining nonsense and never happened. I came up to speed on the tech a bit before learning about the customer, then bumped into it in coreutils a year later and went "oh, I remember that." There was no demand for blake2 that I'm aware of.

[terefang]

The bug here is "glibc broke existing code". The fix is "toybox switches to using the hashes it already has internally".

yes, that would be the most common sense approach !

i am still searching sha3-id ... at least i think i saw one already ... cannot remember where.

sorry for blasting this issue with my braindump about all those use cases, but i had to get it off my mind satisfying my aspergers itch.

[terefang]

if i see that correctly blake2 is used in the argon2 password hashing schema https://www.password-hashing.net/

$ echo -n "password" | ./argon2 somesalt -t 2 -m 16 -p 4 -l 24
Type:           Argon2i
Iterations:     2
Memory:         65536 KiB
Parallelism:    4
Hash:           45d7ac72e76f242b20b77b9bf9bf9d5915894e669a24e6c6
Encoded:        $argon2i$v=19$m=65536,t=2,p=4$c29tZXNhbHQ$RdescudvJCsgt3ub+b+dWRWJTmaaJObG
0.188 seconds
Verification ok

this algorithm is also used by many tools implemented in golang found in containers and small images.

[terefang]

i still could not find a sha3-id for passwd and from what i have read in the mean time i dont think that sha3 will ever be used as password hashing because of implied properties like "slow in software" vs "fast in hardware".

tho i still stand to be corrected on that assumption.

[enh-google]

I'm only familiar with blake2 because over the pandemic I helped out a friend with a project that collapsed into crypto mining nonsense and never happened. I came up to speed on the tech a bit before learning about the customer, then bumped into it in coreutils a year later and went "oh, I remember that." There was no demand for blake2 that I'm aware of.

yeah, it's supported by Android's verified boot -- https://android.googlesource.com/platform/external/avb/+/master/avbtool.py#649 -- for the benefit of armv7 --- so basically dead code in 2023. all current shipping 32-bit devices (Go/TV/Wear) are armv8, so they have hardware sha instructions and don't need something that's cheaper when sofware-only.

(but, if you do decide to add blake2, boringssl does have it[1].)

---

1. blake2b rather than blake2s, which apparently matches the linux kernel.

[landley]

The 2b vs 2s is why it stayed on the todo list: terefang's ID list above had base, y, a, and x variants, and now you're mentioning s. People agree what md5sum, sha1sum, sha256, sha512, and sha3 are. Not so much with this blake2 stuff apparently...

[terefang]

The 2b vs 2s is why it stayed on the todo list: terefang's ID list above had base, y, a, and x variants, and now you're mentioning s. People agree what md5sum, sha1sum, sha256, sha512, and sha3 are. Not so much with this blake2 stuff apparently...

sorry but $2[abyz]$.... are blowfish bcrypt rather than blake2.

only argon2 uses blake2b.

[terefang]

@landley : i saw this on your blog:

Hmmm, can I have something in /etc signaling what kind of password policy to enforce? How about the default format for new passwords is the format of root's password, and failing that sha256?

remembering that redhat had authconfig/authselect --passalgo algo

looks like sources points to

• https://github.com/authselect/authselect
• https://linux.die.net/man/8/authconfig

looks like /etc/login.defs is what you want – https://man7.org/linux/man-pages/man5/login.defs.5.html

ENCRYPT_METHOD (string)
           This defines the system default encryption algorithm for
           encrypting passwords (if no algorithm are specified on the
           command line).

           It can take one of these values: DES (default), MD5, SHA256,
           SHA512. MD5 and DES should not be used for new hashes, see
           crypt(5) for recommendations.

           Note: this parameter overrides the MD5_CRYPT_ENAB variable.

           Note: This only affect the generation of group passwords. The
           generation of user passwords is done by PAM and subject to
           the PAM configuration. It is recommended to set this variable
           consistently with the PAM configuration.

authconfig source gives descrypt|bigcrypt|md5|sha256|sha512|yescrypt as valid values

i suspect that toybox deos not do pam, so me thinks that the last note can be ignored.

[landley]

For the moment I just went with "the default is whatever encryption method root's password used, or sha256 if that doesn't answer it".

Good to know the file's there, and UID_MIN might make more sense than CONFIG_TOYBOX_UID_USR if I could eliminate the config option... except we'd still need a default if the file wasn't there...?

Half this file is... MAILDIR is archaic, UMASK is bashrc, the "ERASECHAR and KILLCHAR are only used on System V machines" implies a comment that's at least 30 years old, SU_NAME is just weird, TTYGROUP seems like an mdev thing or similar, CHFN_RESTRICT seems unlikely to get used much, USERGROUP_ENAB is something I'm unlikely to implement multiple codepaths for, DEFAULT_HOME doesn't seem to need to be selectable, I guess PASS_MAX_DAYS is nice but I haven't implemented that stuff either... and almost everything else seems to be commented out?

Thanks for the heads up though.

Rob

[terefang]

hmm ... sure not to invent new standards, but it may make sense to observe some environmental variable like ENCRYPT_METHOD or PASSWD_ENCRYPT_METHOD or ETC_PASSWD_ENCRYPT_METHOD so someone may put it into /etc/environment for simplicity.

or ETC_LOGIN_DEFS_ENCRYPT_METHOD ?

[landley]

Which would let a normal user change their hash type when they run "passwd" via suid, instead of just root being able to change that system setting...

[terefang]

hmm ... yes ... security problem ... only root should be able to nail this down ... similar to chpasswd

[CameronNemo]

As of this writing, yescrypt is the default password hashing scheme on recent ALT Linux, Arch Linux, Debian 11+, Fedora 35+, Kali Linux 2021.1+, and Ubuntu 22.04+. It is also supported in Fedora 29+, RHEL 9+, and Ubuntu 20.04+, and is recommended for new passwords in Fedora CoreOS.

https://github.com/openwall/yescrypt/blob/e5873f89015c9c7c4ae56e3a6d17cefd47fef239/README#L8C1-L11C70

I agree that "a glibc bug does not obligate toybox to feature creep", but toybox will need to support yescrypt if it wishes to interoperate with the mentioned distros.

[xplshn]

I agree that "a glibc bug does not obligate toybox to feature creep", but toybox will need to support yescrypt if it wishes to interoperate with the mentioned distros.

Seems like this issue is pending close.

Anyways, is there any way to rely on the Musl provided crypt() now that toybox implements its own crypt()?

I'd like to use all of the Musl provided functions instead of those that are built-in just for compatibility with Glibc's bullshit

[landley]

toybox doesn't have its own crypt() replacement yet, it didn't make the release window. Working on it for next release.

I wasn't planning to have multiple selectable implementations here.

[xplshn]

Oh, looks like I have misread the /news page then. My bad.