Include the latest rustdocs (e.g. `cargo doc` output on https://arrow.apache.org/datafusion/)

[alamb]

Is your feature request related to a problem or challenge?

As part of consolidating our documentation and making it better we have been getting the lower level technical documentation about DataFusion into the rustdocs as explained here:

https://arrow.apache.org/datafusion/contributor-guide/architecture.html#architecture

This is good in that keeps the docs close to the code, but it is non ideal in that the docs on crates.io https://docs.rs/datafusion/latest/datafusion/index.html will only reflect released versions and thus will lag behind the latest docs on github

Describe the solution you'd like

I think it is a great idea -- thank you @waynexia

I actually think if we could build those API docs as part of the https://github.com/apache/arrow-datafusion/blob/main/docs build, they would "automatically" get hosted on https://arrow.apache.org/datafusion/

So the idea is if we could get

cd docs
make html

to produce the rustdocs somwhere and link them in somewhere that would be great

More details

https://arrow.apache.org/datafusion/ is published via some ASF mechanism that is similar to github pages

Specifically, this workflow

https://github.com/apache/arrow-datafusion/blob/388f9ec3e7f7c09dac56ee0fe074ca97a6af9d44/.github/workflows/docs.yaml#L12-L64

pushes to the https://github.com/apache/arrow-datafusion/tree/asf-site branch which then gets hosted via this magic yaml:

https://github.com/apache/arrow-datafusion/blob/388f9ec3e7f7c09dac56ee0fe074ca97a6af9d44/.asf.yaml#L48-L52

Describe alternatives you've considered

No response

Additional context

Suggested by @waynexia https://github.com/apache/arrow-datafusion/pull/5962#discussion_r1163748404

[waynexia]

Great! Thank you for sharing those helpful links 🚀 Will try to do this

[alamb]

Looks like there is some more documentation here: https://github.com/apache/arrow-datafusion/tree/main/docs#release-process