Add historic RPC reference

[harding]

@masonicboom has RPC documentation going back to version 0.6.1 and a really cool overview table for them. He's also graciously offered to help bring it to this site. (Thanks!) See this repository for the source.

This issue is for discussing how to do that properly. I'll work on the basic stuff:

• [ ] Copy the doc/ files to this site
• [ ] Quick sed to replace the inline CSS in each file with a ref to this site's regular style sheet
• [ ] Quick style change to blend in with this site's theme
• [ ] Drop master from the table as keeping that updated here will probably generate too much pull request churn
• [ ] Some basic documentation for this site's README about how to use the upstream tool

Note that I plan to use this new table to augment but not replace the current index to the RPC docs, as the table doesn't work well on thin-screen mobiles and also doesn't have a sort order that makes finding particular docs easy (although the ordering is ideal for noticing when RPCs were added/removed, which is nice).

But there are a few more involved issues too:

Fully integrate with site theme, or just cover the basics?

A problem I'm having with the current RPC docs is that the new pages added with each new release is adding about 5 seconds to the site compile time (per release), plus about half a second to the recompile time for an incremental update in preview mode. This is slowing me down on developing other parts of the site---not much so far, but adding up to a thousand pages of new content and continuing to add ~100 new pages per new release has the potential to make developing a real pain.

An idea I have is to make the individual pages of the RPC docs (e.g. for getblock) completely static HTML (i.e. they have no YAML header) so that the site doesn't attempt to render them and just copies them into the output directory (_site) like it does for images, JS, etc.

To do that, I think all we need is a basic header with breadcrumbs back to the main site, then. E.g.:

<a href="../">0.16.2 RPC documentation</a><br>
<h1>getblock</h1>
....

That's a pretty quick change to the format currently used in Mason's files (and easily sed'able), so I'm going to plan on doing that---but I'm open to feedback. If we do that, the URLs currently used for RPC docs on this site will change as they won't be including the RPC section in the URL anymore (which I've been finding inconvenient anyway), so I'll also setup redirects from the old URLs to the new ones for the 0.16.* releases.

What to do when the table grows unreasonably large?

With about 6 total releases per year, this table is likely to grow wider faster than at least my monitors have been lately, so it be nice to have an idea about what to do then. I think figuring this out now is optional as it's not a big deal to just drop rows from the table later if need be.

Feature suggestion: different color when there are documentation differences

Currently the overview table displays binary data: did the RPC exist in x release or not. I think it'd be cool if it also used a different color to show that the documentation in release x is different than release x-1. Even cooler would be if we had a colored diff (word diff?) for the documentation between x-1:x beneath the actual documentation. E.g. this page would have something like this at the bottom:

 "difficulty" : x.xxx, (numeric) The difficulty
 "chainwork" : "xxxx", (string) Expected number of hashes required to produce the chain up to this block (in hex)
+"nTx" : n, (numeric) The number of transactions in the block.
 "previousblockhash" : "hash", (string) The hash of the previous block
 "nextblockhash" : "hash" (string) The hash of the next block

Next steps

• Discussion
• @harding opens PR (maybe WIP) with a preview

CC: @karel-3d

[masonicboom]

This is great! The docs will do more good on bitcoincore.org than my random GitHub site, so if there are upstream changes that make integration simpler, rather than sed hacks, I'm happy to make those.

I made a new light color scheme that is the default when you do make (use MODE=dark make to get the black-and-green scheme). To tweak the colors, check out generate-style.py. I can add CSS classes to table structure if this doesn't play well with bitcoincore.org's CSS rules.

[masonicboom]

Regarding table width: it's probably not necessary to include the docs for each patch release. I'll look into this.

I love the doc diff idea! This would be easy to do if I first dump the docs to e.g. JSON, and then do a transformation pass that renders this to HTML. I think I'll do that. Will enable nicer styling of the doc pages too, if I parse more structure from the docs, like what is an example vs. description, etc...

[harding]

@masonicboom wow, that light color scheme is hot---much nicer than what I would've done. Thanks!

I think @karel-3d would like to have all the minor version releases, but an easy option I think would be to simply have an abridged table for just the major releases and maybe the minor releases for the last two series, plus an unabridged table on a separate page for all of the releases. For displaying a big table, or even a smaller table on a less-wide screen, we might be able to tweak the CSS so that just the filled cells (and not the RPC names) scroll, like this.

Although it sounds like you're considering using JSON on the backend there and you're just using it as a convenient data store rather than part of a JS thing, one note I'd make is that I try to ensure that the frontend user-loaded pages work well even if users have JS disabled. On the backend, however, anything that runs on a reasonable Linux is fine by me. I do like that your code is in Python. :-)

For parsing the upstream docs, here's a brief conversation we had about it before where @MarcoFalke drops some tips that might be helpful.

[masonicboom]

@masonicboom wow, that light color scheme is hot---much nicer than what I would've done.

Thanks ;)

I think @karel-3d would like to have all the minor version releases, but an easy option I think would be to simply have an abridged table for just the major releases and maybe the minor releases for the last two series, plus an unabridged table on a separate page for all of the releases. For displaying a big table, or even a smaller table on a less-wide screen, we might be able to tweak the CSS so that just the filled cells (and not the RPC names) scroll, like this.

Breaking up into different pages sounds smart and simple. I like it.

Although it sounds like you're considering using JSON on the backend there and you're just using it as a convenient data store rather than part of a JS thing, one note I'd make is that I try to ensure that the frontend user-loaded pages work well even if users have JS disabled. On the backend, however, anything that runs on a reasonable Linux is fine by me. I do like that your code is in Python. :-)

Don't worry. The JSON is just an intermediary file format in a pipeline. There is zero JavaScript in this project and I'm keeping it that way. I believe in the separation of JavaScript and Bitcoin.

I'd usually reach for Go, but Python seems to be the de facto scripting language in Bitcoin. Then I saw the current RPC docs are done in Go 😛

For parsing the upstream docs, here's a brief conversation we had about it before where @MarcoFalke drops some tips that might be helpful.

Thanks for the link. Eventually, I might go in and refactor core to generate the docs in a way that's more amenable to parsing. Every wrinkle in the parser points to a way the source could be cleaner.

[masonicboom]

Currently the overview table displays binary data: did the RPC exist in x release or not. I think it'd be cool if it also used a different color to show that the documentation in release x is different than release x-1.

Well, I had some fun here. Thanks again for the idea.

[masonicboom]

The docs are now color coded to show if a call is active, deprecated, or if the doc string changed in that release.

[karelbilek]

Hello, finally some time to look at this :)

1)

The script seems to extract the docs from the source... I extracted them from the CLI version since that seemed easier to me than parsing the source code :) but this scales better, since you don't need to install each version and just look at the git repo

I'd usually reach for Go, but Python seems to be the de facto scripting language in Bitcoin. Then I saw the current RPC docs are done in Go

Haha, I wrote the script in Go since I started to prefer it even for quick scripting - scripting tends to grow anyway and then the types are actually helpful. But Python is indeed more popular in bitcoin world :)

2)

I don't understand the comments about the static HTML in the first post... do you mean to hardcode the CSS, headers, etc into the generating scripts, or do you want to keep the html plaintext?

The delay is unfortunate :( but if we hardcode the CSS etc into the docs, it goes a little against the reason for having jekyll in the first place :D every change will need to be done twice (and the CSS regenerated etc). But it's an OK compromise I guess

3)

Just a random note - if we have all the historical docs, this menu will look terrible and we will probably need to add one more "level" for the minor versions

4)

The table looks great but indeed a little too wide (and tall). Having just major version + subpages to make it less wide is a good idea.

I would indeed prefer a minor releases in docs, if it doesn't make things too complex

It would be nice to group the commands by categories even in the table... but I think the categories were introduced later, right

5)

Eventually, I might go in and refactor core to generate the docs in a way that's more amenable to parsing. Every wrinkle in the parser points to a way the source could be cleaner.

I actually planned on something like that myself :)

=====

ps

I believe in the separation of JavaScript and Bitcoin.

as a contributor to bitcoinjs, I disagree :P but only slightly

[karelbilek]

Slightly related

https://github.com/bitcoin-dot-org/bitcoin.org/issues/2600

I think the "RPC Docs" at bitcoin.org "developer reference" are confusing. They are not versioned and they are outdated. And since bitcoin.org seems to be more implementation-agnostic, I think it does not belong there. (I don't care much about bitcoin.org politics)

[katesalazar]

@masonicboom has RPC documentation going back to version 0.6.1 and a really cool overview table for them. He's also graciously offered to help bring it to this site. (Thanks!) See this repository for the source.

I don't see the point as long as masonicboom updates that site. Just link it somewhere from bitcoincore.org docs pages.

---

Note that I plan to use this new table to augment but not replace the current index to the RPC docs, as the table doesn't work well on thin-screen mobiles and also doesn't have a sort order that makes finding particular docs easy (although the ordering is ideal for noticing when RPCs were added/removed, which is nice).

Conflicts with #867 (disclosure: mine), which is not standalone but branches off from a thought in #687 (not mine, Sjors') who's trusted member like you. Missing a concept NAK there maybe?

---

[lots, really lots, of stuff we coulda quoted here]

As written before, desist. The big complexity of the necessary changes just go blank away, and you'll take proud of being using the Internet in the way it's actually meant to be used. Don't build thrust on the pig until it's the only choice to be taken. I don't think it is (yet). I'd say the proper action here at the moment still is for people to throw work in that other repo, btcrpcapi. Even if it's maybe abandoned (then fork it in bitcoin-core and go ahead).