search

JuliaParallel / MPI.jl

MPI wrappers for Julia
https://juliaparallel.org/MPI.jl/
The Unlicense
376 stars 122 forks source link

[docs] Build with `linkcheck=true` #856

Closed giordano closed 3 weeks ago

giordano commented 1 month ago

Sadly, this doesn't seem to be viable, because there are loads of MPI functions which appear to be undocumented in MPICH and/or OpenMPI websites (https://github.com/JuliaParallel/MPI.jl/actions/runs/10270632053/job/28418784550):

┌ Error: linkcheck 'https://www.mpich.org/static/docs/latest/www3/MPI_Init_thread.html' status: 404.
└ @ Documenter ~/.julia/packages/Documenter/qoyeC/src/utilities/utilities.jl:44
┌ Error: linkcheck 'https://www.mpich.org/static/docs/latest/www3/MPI_Finalize.html' status: 404.
└ @ Documenter ~/.julia/packages/Documenter/qoyeC/src/utilities/utilities.jl:44
[...]
┌ Error: linkcheck 'https://www.open-mpi.org/doc/current/man3/MPI_Accumulate_c.3.php' status: 404.
└ @ Documenter ~/.julia/packages/Documenter/qoyeC/src/utilities/utilities.jl:44
┌ Error: linkcheck 'https://www.mpich.org/static/docs/latest/www3/MPI_Accumulate_c.html' status: 404.
└ @ Documenter ~/.julia/packages/Documenter/qoyeC/src/utilities/utilities.jl:44
┌ Error: linkcheck 'https://www.open-mpi.org/doc/current/man3/MPI_Allgather_c.3.php' status: 404.
└ @ Documenter ~/.julia/packages/Documenter/qoyeC/src/utilities/utilities.jl:44
┌ Error: linkcheck 'https://www.mpich.org/static/docs/latest/www3/MPI_Allgather_c.html' status: 404.
└ @ Documenter ~/.julia/packages/Documenter/qoyeC/src/utilities/utilities.jl:44
┌ Error: linkcheck 'https://www.open-mpi.org/doc/current/man3/MPI_Allgather_init.3.php' status: 404.
└ @ Documenter ~/.julia/packages/Documenter/qoyeC/src/utilities/utilities.jl:44
┌ Error: linkcheck 'https://www.open-mpi.org/doc/current/man3/MPI_Allgather_init_c.3.php' status: 404.
└ @ Documenter ~/.julia/packages/Documenter/qoyeC/src/utilities/utilities.jl:44
┌ Error: linkcheck 'https://www.mpich.org/static/docs/latest/www3/MPI_Allgather_init_c.html' status: 404.
└ @ Documenter ~/.julia/packages/Documenter/qoyeC/src/utilities/utilities.jl:44
┌ Error: linkcheck 'https://www.open-mpi.org/doc/current/man3/MPI_Allgatherv_c.3.php' status: 404.
└ @ Documenter ~/.julia/packages/Documenter/qoyeC/src/utilities/utilities.jl:44
┌ Error: linkcheck 'https://www.mpich.org/static/docs/latest/www3/MPI_Allgatherv_c.html' status: 404.
└ @ Documenter ~/.julia/packages/Documenter/qoyeC/src/utilities/utilities.jl:44
┌ Error: linkcheck 'https://www.open-mpi.org/doc/current/man3/MPI_Allgatherv_init.3.php' status: 404.
└ @ Documenter ~/.julia/packages/Documenter/qoyeC/src/utilities/utilities.jl:44
┌ Error: linkcheck 'https://www.open-mpi.org/doc/current/man3/MPI_Allgatherv_init_c.3.php' status: 404.
└ @ Documenter ~/.julia/packages/Documenter/qoyeC/src/utilities/utilities.jl:44
┌ Error: linkcheck 'https://www.mpich.org/static/docs/latest/www3/MPI_Allgatherv_init_c.html' status: 404.
└ @ Documenter ~/.julia/packages/Documenter/qoyeC/src/utilities/utilities.jl:44
┌ Error: linkcheck 'https://www.open-mpi.org/doc/current/man3/MPI_Allreduce_c.3.php' status: 404.
└ @ Documenter ~/.julia/packages/Documenter/qoyeC/src/utilities/utilities.jl:44
┌ Error: linkcheck 'https://www.mpich.org/static/docs/latest/www3/MPI_Allreduce_c.html' status: 404.
└ @ Documenter ~/.julia/packages/Documenter/qoyeC/src/utilities/utilities.jl:44
┌ Error: linkcheck 'https://www.open-mpi.org/doc/current/man3/MPI_Allreduce_init.3.php' status: 404.
└ @ Documenter ~/.julia/packages/Documenter/qoyeC/src/utilities/utilities.jl:44
┌ Error: linkcheck 'https://www.open-mpi.org/doc/current/man3/MPI_Allreduce_init_c.3.php' status: 404.
└ @ Documenter ~/.julia/packages/Documenter/qoyeC/src/utilities/utilities.jl:44
┌ Error: linkcheck 'https://www.mpich.org/static/docs/latest/www3/MPI_Allreduce_init_c.html' status: 404.
└ @ Documenter ~/.julia/packages/Documenter/qoyeC/src/utilities/utilities.jl:44
┌ Error: linkcheck 'https://www.open-mpi.org/doc/current/man3/MPI_Alltoall_c.3.php' status: 404.
└ @ Documenter ~/.julia/packages/Documenter/qoyeC/src/utilities/utilities.jl:44
┌ Error: linkcheck 'https://www.mpich.org/static/docs/latest/www3/MPI_Alltoall_c.html' status: 404.
└ @ Documenter ~/.julia/packages/Documenter/qoyeC/src/utilities/utilities.jl:44
┌ Error: linkcheck 'https://www.open-mpi.org/doc/current/man3/MPI_Alltoall_init.3.php' status: 404.
└ @ Documenter ~/.julia/packages/Documenter/qoyeC/src/utilities/utilities.jl:44
┌ Error: linkcheck 'https://www.open-mpi.org/doc/current/man3/MPI_Alltoall_init_c.3.php' status: 404.
└ @ Documenter ~/.julia/packages/Documenter/qoyeC/src/utilities/utilities.jl:44
┌ Error: linkcheck 'https://www.mpich.org/static/docs/latest/www3/MPI_Alltoall_init_c.html' status: 404.
└ @ Documenter ~/.julia/packages/Documenter/qoyeC/src/utilities/utilities.jl:44
┌ Error: linkcheck 'https://www.open-mpi.org/doc/current/man3/MPI_Alltoallv_c.3.php' status: 404.
└ @ Documenter ~/.julia/packages/Documenter/qoyeC/src/utilities/utilities.jl:44
┌ Error: linkcheck 'https://www.mpich.org/static/docs/latest/www3/MPI_Alltoallv_c.html' status: 404.
└ @ Documenter ~/.julia/packages/Documenter/qoyeC/src/utilities/utilities.jl:44
┌ Error: linkcheck 'https://www.open-mpi.org/doc/current/man3/MPI_Alltoallv_init.3.php' status: 404.
└ @ Documenter ~/.julia/packages/Documenter/qoyeC/src/utilities/utilities.jl:44
┌ Error: linkcheck 'https://www.open-mpi.org/doc/current/man3/MPI_Alltoallv_init_c.3.php' status: 404.
└ @ Documenter ~/.julia/packages/Documenter/qoyeC/src/utilities/utilities.jl:44
[...]

For example, documentation for MPI_Init_thread in MPICH seems to have been removed between v4.0 and v4.1 (see also https://www.mpich.org/static/docs/v4.0/ vs https://www.mpich.org/static/docs/v4.1/). Edit: missing MPICH manpages will be fixed by https://github.com/pmodels/mpich/pull/7107. See also https://github.com/pmodels/mpich/issues/7089

giordano commented 1 month ago

Actually... I think OpenMPI website moved to https://docs.open-mpi.org/en/main/man-openmpi/man3/$fname.3.html

giordano commented 1 month ago

With latest changes, missing docstrings reduced to

 ┌ Error: linkcheck 'https://www.mpich.org/static/docs/v4.0/www3/MPI_Status_c2f.html' status: 404.
└ @ Documenter ~/.julia/packages/Documenter/qoyeC/src/utilities/utilities.jl:44
┌ Error: linkcheck 'https://www.mpich.org/static/docs/v4.0/www3/MPI_Status_f2c.html' status: 404.
└ @ Documenter ~/.julia/packages/Documenter/qoyeC/src/utilities/utilities.jl:44
┌ Error: linkcheck 'https://www.mpich.org/static/docs/v4.0/www3/MPI_Type_create_f90_complex.html' status: 404.
└ @ Documenter ~/.julia/packages/Documenter/qoyeC/src/utilities/utilities.jl:44
┌ Error: linkcheck 'https://www.mpich.org/static/docs/v4.0/www3/MPI_Type_create_f90_integer.html' status: 404.
└ @ Documenter ~/.julia/packages/Documenter/qoyeC/src/utilities/utilities.jl:44
┌ Error: linkcheck 'https://www.mpich.org/static/docs/v4.0/www3/MPI_Type_create_f90_real.html' status: 404.
└ @ Documenter ~/.julia/packages/Documenter/qoyeC/src/utilities/utilities.jl:44

Edit: all autogenerated links have been fixed by #859.

giordano commented 1 month ago

Last error, after #854:

┌ Error: linkcheck 'https://www.intel.com/content/www/us/en/developer/tools/oneapi/mpi-library.html' status: 403.
└ @ Documenter ~/.julia/packages/Documenter/qoyeC/src/utilities/utilities.jl:44

but I'm confused, locally I get:

$ curl -I 'https://www.intel.com/content/www/us/en/developer/tools/oneapi/mpi-library.html'
HTTP/2 200
content-type: text/html;charset=utf-8
server: Apache
x-dispatcher: dispatcher2uswest1-b80
x-vhost: publish
last-modified: Wed, 07 Aug 2024 13:09:27 GMT
etag: W/"30f33-61f179eb85c9b-gzip"
expires: Wed, 07 Aug 2024 21:40:56 GMT
cache-control: max-age=0, no-cache
pragma: no-cache
date: Wed, 07 Aug 2024 21:40:56 GMT
set-cookie: detected_bandwidth=LOW; path=/; domain=.intel.com; secure; HttpOnly
set-cookie: src_countrycode=gb; path=/; domain=.intel.com; secure; HttpOnly
alt-svc: h3=":443"; ma=93600
content-security-policy: frame-ancestors 'self' https://premiersupport.intel.com https://c0.avaamo.com *.intel.com; object-src 'self';
x-grn: 0.8d3e1202.1723066856.1047ccba
x-frame-options: SAMEORIGIN
access-control-allow-origin: *
x-xss-protection: 1; mode=block
strict-transport-security: max-age=31536000 ; preload
x-content-type-options: nosniff

I hope they aren't blocking connections from GitHub-hosted runners...

giordano commented 1 month ago

Ah, the problem is the user agent used by Documenter:

$ curl --user-agent "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36" -sI 'https://www.intel.com/content/www/us/en/developer/tools/oneapi/mpi-library.html'
HTTP/2 403
server: AkamaiGHost
mime-version: 1.0
content-type: text/html
content-length: 460
expires: Wed, 07 Aug 2024 21:46:56 GMT
cache-control: max-age=0, no-cache
pragma: no-cache
date: Wed, 07 Aug 2024 21:46:56 GMT
set-cookie: detected_bandwidth=LOW; path=/; domain=.intel.com; secure; HttpOnly
set-cookie: src_countrycode=gb; path=/; domain=.intel.com; secure; HttpOnly
alt-svc: h3=":443"; ma=93600
content-security-policy: frame-ancestors 'self' https://premiersupport.intel.com https://c0.avaamo.com *.intel.com; object-src 'self';
x-grn: 0.8d3e1202.1723067216.1050300f
x-frame-options: SAMEORIGIN
access-control-allow-origin: *
x-xss-protection: 1; mode=block
strict-transport-security: max-age=31536000 ; preload
x-content-type-options: nosniff

Edit: issue reported at https://github.com/JuliaDocs/Documenter.jl/issues/2557