Re-generate docs for dependents on generic-array v0.14

newpavlov commented 5 months ago

A big number of RustCrypto crates was affected by https://github.com/rust-lang/rust/issues/120983 The bug was fixed, but many crates are still affected since no new releases were uploaded since. For example, see docs for sha2::Digest. Since we are currently preparing the next release cycle, we do not plan to publish new patch versions for many affected crates.

Would it be possible to re-generate docs for all dependents of generic-array v0.14? The number of affected crates is too big to create this list manually.

syphar commented 5 months ago

Some first notes before I can dig deeper:

first of all: when we have a list of crate names & exact versions, then we can totally trigger a rebuild for them.

For generating that list, our docs.rs database is not very well fitted. While we store the dependencies of each release, it's just a JSON blob so (right now) can't be queried efficiently.

So for generating that list we would probably look at the crates.io API or database.

One other thing to keep in mind: we would not only have to check exact matches but also semver matches. I'm not sure if the crates.io reverse dependency check (which powers the "dependends" tab) covers this.

syphar commented 5 months ago

I think the crates.io reverse dependencies list is the right point to start.

syphar commented 5 months ago

Using the crates.io API I generated the list of releases, @newpavlov can you confirm that the list is fine?

Then I'll trigger the rebuilds.

Releases that depend on any 0.14 version, taking semver into account

``` aead 0.5.2 aligned-array 1.0.1 aligned-cmov 2.3.0 aloha 0.1.0 amiitool-rs 0.2.0 amora-rs 0.2.1 antimatter 1.0.5 arecibo 0.1.1 assemble-core 0.2.0 at-cryptoauth 0.4.0 atuin-client 18.3.0 ax_banyan 0.18.0 babyjubjub-rs 0.0.11 bandsocks 0.2.1 bandsocks-sand 0.2.0 banyan 0.17.1 bbachain-frozen-abi 0.1.1 bitwarden-crypto 0.5.0 block-buffer 0.10.4 block-padding 0.3.3 bloock-babyjubjub-rs 0.0.11 bulletproof-kzen 1.2.1 cairo-vm 0.9.3 cartesi-solana 0.3.2 cbe-frozen-abi 1.15.0 cbe-sdk 1.15.0 cbrzn-ethers-core 1.0.2 cdbc-mysql 0.1.22 centipede 0.3.1 cess-hashers 0.1.0 cess-proofs 0.1.2 cess-sp-core 0.1.2 cess-sp-porep 0.1.2 cess-sp-post 0.1.2 chamber-cli 0.2.0 chamber-core 0.2.0 citadel_pqcrypto 0.9.0 cnfgen 0.4.0 coins-core 0.11.1 conundrum 0.1.0 cprf 0.1.4 cry 0.1.5 crypticy 0.2.2 crypto-bigint 0.5.5 crypto-common 0.1.6 crypto-mac 0.11.1 crypto_secretbox 0.1.1 csrf 0.4.1 ct-merkle 0.1.0 curv-kzen 0.10.0 curv-lsc 0.1.2 daence 0.0.0 dbl 0.3.2 deterministic-pgp-keys 0.5.0 detour 0.8.1 detour2 0.9.0 detour3 0.1.0 dex-connector 1.1.2 dimensioned 0.8.0 drogue-tls 0.6.0 dryoc 0.5.3 easy-hex 1.0.0 elliptic-curve 0.13.8 elliptic-curve-flow 1.0.0 em7180 0.2.0 embassy-net 0.4.0 embedded-tls 0.17.0 eridani 0.1.0 etf-crypto-primitives 0.2.3 eth-stealth-addresses 0.1.0 ethers-core 2.0.14 etoml 0.2.1 everscale-crypto 0.2.1 everscale-network 0.5.4 fastcrypto 0.1.8 fdups 0.2.1-dev ferveo-common-pre-release 0.1.1 ferveo-pre-release 0.3.0 field-encryption 0.0.1 filecoin-hashers 13.1.0 filecoin-proofs 18.1.0 fire-crypto 0.4.3 fixed-polynomial 0.1.0 flize 4.2.3 foco 0.1.4 forest_cid 0.3.0 fundsp 0.18.1 gemachain-frozen-abi 1.8.2 gemachain-sdk 1.8.2 generic-arrayvec 0.4.0 generic-bytes 0.2.2 generic-bytes-derive 0.2.2 generic-ec-core 0.2.0 generic-ec-zkp 0.4.0 github-webhook-extract 0.1.0 gost-modes 0.5.0 gp2y0e02b 0.2.1 gyuvl53l0x 0.3.0 hashwires 0.1.0 hawkbit 0.6.0 hctr2 0.2.0 hddwiper 2.0.0 hmac-drbg 0.3.0 homestar-workspace-hack 0.1.0 hpke 0.11.0 hpke_pq 0.10.1 ibc-relayer 0.28.0 inout 0.1.3 iota-crypto 0.23.1 iq-crypto 0.0.1 iso8583_rs 0.1.10 jsonwebkey 0.3.5 jupyter 0.2.1 keepass-db 0.0.2 keeshond 0.27.0 keeshond_datapack 0.13.0 kem 0.2.0 kinode 0.5.0 libsfasta 0.3.4 linera-base 0.11.3 linera-views 0.11.3 lioness-rs 0.1.0 littlefs2 0.4.0 lldap_auth 0.3.0 lms-signature 0.0.1 lnk-thrussh 0.33.5 lorawan 0.8.0 lorawan-device 0.12.1 lorawan-encoding 0.6.2 lpc55-hal 0.3.0 lurk 0.3.1 mas-jose 0.9.0 mas-keystore 0.9.0 matrixmultiply_mt 0.2.1 mc-oblivious-map 2.3.0 mcan 0.5.0 median 0.3.2 melange 0.1.0 melodies-blake2 0.1.0 mental-poker 0.1.0 minimal-ed448 0.4.0 miraland-frozen-abi 1.18.3 miraland-sdk 1.18.3 mla 1.4.0 moshudp 0.1.4 mstickerlib 0.3.1 nash-mpc 1.2.4 navajo 0.0.4 neptune 13.0.0 nested-ref 0.1.0 nextpass 0.4.1 nimue 0.1.0 nucypher-core 0.14.0 numeric-array 0.5.2 nym-crypto 0.4.0 opaque-ke 2.0.0 parity-scale-codec 3.6.12 passdata 0.0.2 passkey-authenticator 0.2.0 pasta-tokens 0.1.0-rc.7 pasteit 0.0.1-alpha pastry 0.0.1 pgp 0.13.0 poseidon-client 0.6.2 precrypt 0.3.5 proximity 0.1.1 pykrete-jsonwebkey 0.4.1 rac 1.3.4 radicle-keystore 0.2.0 rattler_digest 0.19.4 retour 0.3.1 ricq-core 0.1.19 ridl 0.7.0 ring-compat 0.8.0 rlwe 0.1.0 robust-geo 0.1.7 rops 0.1.3 rsfbclient-rust 0.24.0 rubedo 0.5.3 rudolfs 0.3.7 russh 0.43.0 rust-kbkdf 1.1.0 rustmacaroon 0.1.2 rusty_paserk 0.4.0 rwarden_crypto 0.0.1 safecoin-frozen-abi 1.14.17 safecoin-sdk 1.14.17 same-content 0.1.10 scromble 2.0.1 scsys-crypto 0.1.40 sec1 0.7.3 secp256kfun_k256_backend 2.0.1 secret-service 4.0.0 secure-session 0.4.0 seeed-erpc 0.1.1 sha1collisiondetection 0.3.4 sha2-derive 0.1.2 shoes 0.1.2 shotover 0.3.1 sigma_fun 0.7.0 signal-backup-decode 0.2.3 skipjack_rs 0.1.0 sliding_window 0.1.2 slip-10 0.4.0 sn_curv 0.10.1 solana-frozen-abi 2.0.0 solana-sdk 2.0.0 solomka-sdk 1.14.20 soroban-env-host 21.1.0 spaghettinuum 0.2.2 spartan2 0.1.0 spmc_ring 0.1.0 spms_ring 0.1.4 sqlx-core-guts 0.6.0 sqlx-core_wasi 0.6.2 sqlx-mysql 0.7.4 srp 0.6.0 ssd1362 0.1.0 stack_dst 0.8.1 static-dh-ecdh 0.1.1 storage-proofs-core 18.1.0 storage-proofs-porep 18.1.0 storage-proofs-post 18.1.0 storage-proofs-update 18.1.0 tari_utilities 0.7.0 taro 0.1.0 teleporter 0.10.8 tetsy-scale-codec 2.0.1 thrussh 0.35.4 tink-aead 0.2.5 tink-signature 0.2.5 tiny-multihash 0.5.0 tiny-sdk 1.15.0 topq 0.2.0 trezor-crypto 0.2.5 trussed 0.1.0 tsproto 0.2.0 tsproto-types 0.1.0 ukf 0.1.2 umbral-pre 0.11.0 undr 0.2.0 universal_wallet 0.6.1 usig 0.11.0 veb-tree 0.1.0 vented 0.11.7 verilot 0.1.0 vimdecrypt 0.1.3 voprf 0.5.0 vru-noise 1.7.1 vru-sphinx 1.5.0 waffles-solana-frozen-abi 1.16.0 waffles-solana-sdk 1.16.0 wasefire 0.5.0 wasefire-board-api 0.6.0 wasefire-scheduler 0.3.0 wasefire-stub 0.1.3 welds-sqlx-mssql 1.7.0 wio_terminal 0.7.0 wormhole-bridge-terra 0.1.0 yaap 0.0.2 yafo 0.1.1 yap2p 0.1.2 ```

newpavlov commented 5 months ago

@syphar Unfortunately, the list captures only crypto-common. Crates which define algorithm traits (e.g. digest) use generic-array re-exported from crypto-common. Finally, implementation crates (e.g. sha2) implement the traits and thus use generic-array which was re-exported from digest, which in turn uses re-exported version from crypto-common.

Ideally, we need to re-generate docs for crates which expose GenericArray in their public API and which were published in a certain date range. But it looks too complex for resolving this relatively minor issue.

If we are fine with resolving this issue only for RustCrypto crates, then one option is to re-generate docs only for crates which have a RustCrypto group owner (e.g. github:rustcrypto:hashes). Here is a list of relevant groups: AEADs, block-ciphers, crypto-bigint, elliptic-curves, formats, hashes, JOSE, KDFs, KEMs, key-wraps, MACs, nacl-compat, PAKEs, password-hashes, RSA, signatures, sponges, SSH, stream-ciphers, traits, utils.

syphar commented 5 months ago

@syphar Unfortunately, the list captures only crypto-common. Crates which define algorithm traits (e.g. digest) use generic-array re-exported from crypto-common. Finally, implementation crates (e.g. sha2) implement the traits and thus use generic-array which was re-exported from digest, which in turn uses re-exported version from crypto-common.

🤦 I didn't think about dependencies of dependencies.

I'm not sure if doing the crates.io query recursively would even finish. And with semver, would take quite some time to figure out correctly.

Ideally, we need to re-generate docs for crates which expose GenericArray in their public API and which were published in a certain date range. But it looks too complex for resolving this relatively minor issue.

Sadly that kind of query is not possible yet (-> #1467).

If we are fine with resolving this issue only for RustCrypto crates, then one option is to re-generate docs only for crates which have a RustCrypto group owner (e.g. github:rustcrypto:hashes). Here is a list of relevant groups: AEADs, block-ciphers, crypto-bigint, elliptic-curves, formats, hashes, JOSE, KDFs, KEMs, key-wraps, MACs, nacl-compat, PAKEs, password-hashes, RSA, signatures, sponges, SSH, stream-ciphers, traits, utils.

fine with me for now.

Can you confirm that list? I checked the releases for the crates for these owners, where the build-status is failed.

releases for these owners with failed build status

SQL statement used (for me)

``` select o.login as owner , c.name as crate, rr.version from owners as o inner join owner_rels as r on r.oid = o.id inner join crates as c on r.cid = c.id inner join releases as rr on c.id = rr.crate_id where o.login like 'github:rustcrypto:%' and rr.rustdoc_status = false order by c.name, rr.version ```

newpavlov commented 5 months ago

You don't need the "build-status is failed" check (but you need to filter pre versions). The problematic docs are generated properly, they just don't have links generated for GenericArray. For example, see the Digest link in the OP. In other words, the list should contain sha2 v0.10.8, digest v0.10.7, block-buffer v0.10.4, and crypto-common v0.1.6.

syphar commented 5 months ago

(but you need to filter pre versions)

which?

without the build status filter I'm getting 2676 releases, which we could rebuild, but I would prefer having a filter.

newpavlov commented 5 months ago

For example, with sha2 we need to rebuild v0.10.8, and ignore v0.11.0-pre* versions. I think we can rebuild only the highest non-pre version for each crate. It would mean that sha2 v0.10.7 will be still affected, but probability of users reading its docs is much lower.

syphar commented 3 months ago

coming back to open issues.

I can queue rebuilds for any list of (name, version).

To safe us some back and forth around the list I would propose you generate that list, for example based on the crates.io database dump, which has all crates & their dependencies.

Or should we just wait for crate authors to ping us?

newpavlov commented 2 months ago

Sorry for the late reply! Here is a manually compiled list. It's far from being complete, but should cover most visible crates.

``` aead 0.5.2 cipher 0.4.4 crypto-common 0.1.6 digest 0.10.7 password-hash 0.5.0 universal-hash 0.5.1 ascon 0.2.0 belt‑hash 0.1.1 blake2 0.10.6 fsb 0.10.4 gost94 0.10.1 groestl 0.10.1 jh 0.1.0 k12 0.3.0 md2 0.10.2 md4 0.10.2 md5 0.10.6 ripemd 0.1.3 sha1 0.10.6 sha2 0.10.8 sha3 0.10.8 shabal 0.4.1 skein 0.1.0 sm3 0.4.2 streebog 0.10.2 tiger 0.2.1 whirlpool 0.10.4 aes 0.8.4 aria 0.1.0 belt-block 0.1.2 blowfish 0.9.1 camellia 0.1.0 cast5 0.11.1 cast6 0.1.0 des 0.8.1 idea 0.5.1 kuznyechik 0.8.2 magma 0.9.0 rc2 0.8.1 rc5 0.0.1 serpent 0.5.1 sm4 0.5.1 speck-cipher 0.0.1 threefish 0.5.2 twofish 0.7.1 chacha20 0.9.1 hc-256 0.5.0 rabbit 0.4.1 rc4 0.1.0 salsa20 0.10.2 ghash 0.5.1 poly1305 0.8.0 polyval 0.6.2 aes-gcm-siv 0.11.1 aes-gcm 0.10.3 aes-siv 0.7.0 ascon-aead 0.4.2 ccm 0.5.0 chacha20poly1305 0.10.1 deoxys 0.1.0 eax 0.5.0 mgm 0.4.6 belt-mac 0.1.0 cbc-mac 0.1.1 cmac 0.7.2 hmac 0.12.1 pmac 0.7.1 hkdf 0.12.4 concat-kdf 0.1.0 ```

syphar commented 2 months ago

I queued the rebuilds.

Versions that didn't exist:

belt‑hash 0.1.1
fsb 0.10.4
md5 0.10.6

syphar commented 2 months ago

progress here: https://docs.rs/releases/queue

newpavlov commented 2 months ago

Versions that didn't exist:

Fixed list: md-5 0.10.6, fsb 0.1.3.

belt-hash 0.1.1 should exist.

syphar commented 2 months ago

queued rebuilds for all three

newpavlov commented 2 months ago

Thank you!

rust-lang / docs.rs

Re-generate docs for dependents on generic-array v0.14 #2528