[DX][UX] A ritual sacrifice necessary to make a directive parse RST

[webknjaz]

clickbait! I hope this title catches some attention :)

But, really... I want to raise a concern about the developer experience of some APIs, in particular, those that are supposed to help build Sphinx extensions.

I've built a few extensions in the past and regularly hit some roadblocks along the way. Some of the walls I hit made me give up and retry in half a year or more in order to successfully fight the corresponding APIs. I must admit that part of the problems come from parts of docutils leaking into the Sphinx's public interfaces and forcing the developers to read the source of both projects to figure things out because of the poor docs.

This is how it feels right now:

Some of the problems can be hotfixed short-term by improving the docs and adding more illustrative examples. But the better long-term solution would be implementing better, ~user~developer-friendly interfaces.

Let me tell you about my latest experience. The recent (re-)discovery is sphinx.util.nodes.nested_parse_with_titles(). The doc promises a painless way to take some RST and turn in into docutils nodes that can be returned from a directive. I think I tried it about a year ago and got away just constructing some nodes manually. This time, when I needed to do something similar but more complex, I was smarter and went to read the source of the sphinx's include directive that turns out to wrap docutils' include directive and ended up seeing that it uses state_machine.insert_input() and knowing that Directive has it exposed on the object I just went ahead and used it:

self.state_machine.insert_input(
    statemachine.string2lines(rst_source),
    '[my custom included source]',
)
return []

Later, @ewjoachim figured out how to actually make sphinx.util.nodes.nested_parse_with_titles() work (https://github.com/ansible/pylibssh/pull/119/files#diff-93857a1c2f2d5628aadfb443d70a87eeR111-R121). But this is admittedly a less maintainable/readable approach that is far from being obvious even to experienced devs (https://twitter.com/Ewjoachim/status/1289323468270395393).

I'm not going to add extra examples, because I don't remember what other problems were exactly, I only have this feeling left that we can do better!

So in order for this to be not another pointless rant, let me suggest how this specific interface could be improved.

First of all, there should be function that just takes input and just returns an output. That's it! No side-effects. Passing a variable to a function only for it to be mutated by reference is an ancient technique coming from C projects. They simply didn't have much flexibility: the return value was usually some return code of a flag representing the success or the failure of an invocation. In Python, we can do better, we raise exceptions for problems and just return the results, it's that simple.

Second, creating a temporary fake node only to discard it two lines later, extracting the children messes up the readability too. It should be created if the need to use it arises, not just because some API cannot return a list...

Finally, here's an API I'm having in my mind. How about extending Directive with this method:

def convert_rst_to_nodes(self, rst_source: str) -> List[nodes.Node]:
    """Turn an RST string into a node that can be used in the document."""
    node = nodes.Element()
    node.document = self.state.document
    nested_parse_with_titles(
        state=self.state,
        content=statemachine.ViewList(
            statemachine.string2lines(rst_source),
            source='[custom RST input]',
        ),
        node=node,
    )
    return node.children

And then, it could be used as

def run(self) -> List[nodes.Node]:
    rst_source = ...
    return self.convert_rst_to_nodes(rst_source)

[chrisjsewell]

I must admit that part of the problems come from parts of docutils leaking into the Sphinx's public interfaces and forcing the developers to read the source of both projects to figure things out because of the poor docs.

Hey @webknjaz fancy seeing you here 😄 This is a bit of a tangent, but I was just searching issues mentioning docutils and this come up. Basically, I think your quote "we can do better" very much applies to that package; its documentation is sub-par at best, and there are much needed improvements (plus its not even on version 1 after 18 years!). An example of this is the fact that sphinx itself has to monkey-patch docutils code:

https://github.com/sphinx-doc/sphinx/blob/57af8288826f9a37cb3ba3008fb36278ecbae81f/sphinx/util/docutils.py#L164

and when you mention "No side-effects" I very much agree, and found issues in this when writing myst-parser and when trying to implement an asynchronous language-server: rst-language-server (because nodes/roles/directives are stored as global dictionaries, which are not thread safe)

IMO a first step would be to get docutils to actually move to GitHub, so it was easier to contribute and and improve things from the ground up. At present docutils have no plans to do this, see https://sourceforge.net/p/docutils/feature-requests/58/ and the duplicate https://sourceforge.net/p/docutils/bugs/372/, but is notable the small amount of comments these issues have received. Perhaps a concerted in-flux of "+1" comments by the community on this issue, may "help" docutils reconsider this stance 😬

cc @choldgraf @mmcky @asmeurer

[choldgraf]

I'm a big +1 on both of these :-)

A few thoughts:

• First, I wanna recognize how hard it is to keep a complex machine like Sphinx documented in its totality with very limited resources - thanks to the Sphinx community for continuing to work on this and make improvements 👍
• That said, I think a good first step is figuring out all of these "hacks" that developers have figured out within Sphinx, and document (some) of them in the documentation. The documentation should make it clear what's the best / easiest pathways to getting common things done in Sphinx.*
• I think beyond that, I also agree in streamlining some of the underlying components of Sphinx so that they're easier to extend/reuse/etc (though the challenge here is that API changes are always disruptive)
• For docutils I also completely agree, though I don't know who is leading efforts there so I'm not sure who is the person to make a case to (beyond opening an issue). Perhaps one path forward is to reach out to the PSF?

* Note that some of this should come from us, IMO. As developers that also use Sphinx, I think we're in the best position to know where the Sphinx documentation is lacking, and also often have the right insider knowledge to know how the documentation could be improved.

[asmeurer]

Yes, docutils feels like abandonware (I don't know if it's actually abandoned, but if feels like that). This is not good for a project that is critical infrastructure in the Python ecosystem.

I'm not entirely clear what parts of Sphinx/docutils this is suggesting to change, but I wonder if it would help with projects like sphinx-math-dollar (https://github.com/sympy/sphinx-math-dollar). Right now, it has to dig very deep into docutils in order to do what it does because the syntax is fundamentally incompatible with RST (it has to pick up what is inside of dollar signs before the RST parser does). Even now, it doesn't work properly with substitutions (https://github.com/sympy/sphinx-math-dollar/issues/16).

Sorry if this isn't actually related to this issue. I do agree that the way docutils manages its AST as a mutable datastructure is very clunky, and there are much more modern and maintainable ways of managing ASTs.

[chrisjsewell]

First, I wanna recognize how hard it is to keep a complex machine like Sphinx documented in its totality with very limited resources

Indeed, I have no major issues with Sphinx itself. Obviously packages can always be improved, but the rate of progress, given the resources, is great thanks 😄.

For docutils though, not a fan lol. I would be interested in knowing how much dialogue the sphinx guys have with them? Given that, it appears to me at least, the continued existence of docutils is largely due to its use by sphinx, it would make so much more sense for it to live under this GitHub organization. The maintenance burden would obviously be a factor, but the scope for integration and bottom-up fixes would be massive.

[webknjaz]

Basically, I think your quote "we can do better" very much applies to that package; its documentation is sub-par at best, and there are much needed improvements (plus its not even on version 1 after 18 years!).

I know, right? Although, I wanted to scope this to Sphinx specifically because they don't have ultimate power over docutils anyway. My point is that good abstractions would result in a better UX. Also, I'm not aiming to find whose fault this is but to have a more productive resolution instead.

IMO a first step would be to get docutils to actually move to GitHub, so it was easier to contribute and and improve things from the ground up. At present docutils have no plans to do this, see sourceforge.net/p/docutils/feature-requests/58 and the duplicate sourceforge.net/p/docutils/bugs/372, but is notable the small amount of comments these issues have received. Perhaps a concerted in-flux of "+1" comments by the community on this issue, may "help" docutils reconsider this stance

I understand the sentiment here but shaming docutils for not doing what people want is not going to be helpful. I'm pretty sure when the project started what they did made perfect sense. In fact, we all know that maintaining things is hard so putting this sort of pressure is not productive. I'd even go as far as saying that the maintainers have right for their opinion. The open-source way has always been to make a fork if you want to do something that the original project doesn't intend to do.

It looks like if Sphinx team (or some other concerned party) were to make some docutils-ng fork, it'd stand a way better chance of getting nicer APIs and maintainability in general.

[ewjoachim]

I find it very positive that this discussion is happening. I'm wondering what the way forward is:

• Documenting things ?
• Creating a library on top of docutils that provides a sane abstraction layer ?
• Creating a new library replacing docutils for the needs of Sphinx ? (in both case, the compatibility requirements will be very complex)

In short, what do you imagine needs to be done so that, in a plausible future not so far away, people don't have to make a ritual sacrifice in order to make new sphinx (/RST) features ? This could be as "simple" as an extensive documentation project within docutils with a lot of examples, code snippets, and written for people who don't already know the lib.

[mgeier]

To get the attention of the docutils devs, someone should probably mention this discussion on their mailing list: https://sourceforge.net/projects/docutils/lists/docutils-users?

[ewjoachim]

I've sent an email but it doesn't appear yet, nor did I received an ACK, so I don't know if I'm just greylisted, if something went wrong or if my mail was discarder as spam. Time will tell.

[tk0miya]

AFAIK, the maintainers of docutils have surely alive. Indeed, they did not react soon, but not dead. I have experience in posting patches to their list and got merged. So it would be better to send patches to docutils team instead of forking.

[alan-isaac]

The docutils developers have been very willing to incorporate patches. The new rst2html5 writer is very good. Some of the critical energy displayed in this thread would be better spent in offering patches. The development team is very small.

I don't think the developers have ever considered the use of Subversion rather than git to be a serious objection rather than a statement of tastes, especially given the availability of git-svn. The superiority of git for such a small project is certainly contestable. (E.g., https://svnvsgit.com/)

[choldgraf]

Just a note that I just saw the email about this show up on the docutils mailing list.

re: offering patches, I think one of the main topics of this thread is "there are people who would love to contribute patches but doing so via subversion and sourceforge makes it much harder to do so, generally hard enough that it just doesn't happen".

[alan-isaac]

Makes it harder ... how? Isn't it really more than a matter of taste? (I'm open to the possibility that there is a real issue here, but I rather doubt it; see the link I included.) Why is git-svn not good enough for those who find Subversion annoying?

[chrisjsewell]

I've sent an email

Thanks @ewjoachim I see it, I don't think this would make them reconsider though.

but shaming docutils for not doing what people want is not going to be helpful

I don't see it as shaming, I just think there needs to be more feedback to them. As I mentioned, there is only a small amount of people that have actually commented/+1'd moving to GitHub. I can empathise as a developer, that if one person raises an issue which I don't personally see the importance of, it will be low on the TODO list, but if many more people start agreeing on the issue it certainly makes me reconsider.

there are people who would love to contribute patches but doing so via subversion and sourceforge makes it much harder to do so

Exactly. From what I can see, in the whole of 2020, there has been precisely one closed patch and five others opened:

• https://sourceforge.net/p/docutils/patches/search/?q=status%3Aclosed-wont-fix+or+status%3Aclosed-rejected+or+status%3Aclosed-out-of-date+or+status%3Aclosed-accepted+or+status%3Aclosed+or+status%3Aclosed-fixed
• https://sourceforge.net/p/docutils/patches/search/?q=%21status%3Aclosed-wont-fix+%26%26+%21status%3Aclosed-rejected+%26%26+%21status%3Aclosed-out-of-date+%26%26+%21status%3Aclosed-accepted+%26%26+%21status%3Aclosed+%26%26+%21status%3Aclosed-fixed

That's not healthy for such a prominent open source package. It's not that it can't be done, it's that it takes a lot more energy than necessary

The development team is very small.

This is exactly the reason why opening up the code to more contributors would help them. I don't think it's too controversial to say that GitHub/GitLab are far superior collaboration environments, which many more people are comfortable with.

This could be as "simple" as an extensive documentation project within docutils with a lot of examples, code snippets, and written for people who don't already know the lib.

I think this would be a great first step, as well as external documentation, more/improved docstrings, function typing etc

Finally, here's an API I'm having in my mind. How about extending Directive with this method:

Just with my MyST bias, I would ask that a higher level abstraction doesn't specifically reference rST (e.g. convert_rst_to_nodes), since the actual AST is source independent 😁

[choldgraf]

Sure - individually it is a matter of taste, but if you're building teams and communities around tools, then the choice of tool will have a huge impact on the ability to grow a community around it. Each extra step a developer must take in order to contribute drastically reduces their likelihood of doing so. This is particularly true for dev tooling - it is significantly harder to get people to engage with open source projects if those people need to learn a new tool chain to do so.

The majority of the modern open source community doesn't use subversion or sourceforge, they use git and github/gitlab. If one of the people from this broader community (like us) wants to contribute to docutils, they need to learn a new CLI tool (or install a meta-tool like git-svn to figure it out), they also need to learn a new UI for collaborative coding (SourceForge) that has a lot of UX elements that differ from what people are used to in GH/GL. I suspect that these alone are enough of a barrier to prevent most people from offering patches.

As @chrisjsewell says - the whole point is to ease the ability of a broader community to engage with docutils, and to potentially increase the number of people that help maintain it. In its current state on sourceforge, I don't think that docutils will likely grow past its current group of maintainers.

[chrisjsewell]

Makes it harder ... how? Isn't it really more than a matter of taste? (I'm open to the possibility that there is a real issue here, but I rather doubt it; see the link I included.) Why is git-svn not good enough for those who find Subversion annoying?

@alan-isaac well the maintainers of Python itself have something to say on this matter 😉 https://snarky.ca/the-history-behind-the-decision-to-move-python-to-github/

[tk0miya]

Why do you discuss about docutils topic in Sphinx project? There are no maintainers of docutils here. If you'd like to change the docutils project, let's discuss in docutils-sig.

[chrisjsewell]

Why do you discuss about docutils topic in Sphinx project?

Because the status of docutils has a very direct impact on Sphinx, and this issue in particular. Also any discussion started in docutils-sig I felt would fall on deaf ears. For example, there are already these issues with no replies:

• https://sourceforge.net/p/docutils/feature-requests/58/
• https://github.com/docutils-mirror/docutils/issues/2

[tk0miya]

What is a goal of this discussion? We don't have an ownership of docutils project. So we can't move the repository to GitHub. I still don't understand why do you discuss it here. It is out of topic.

[choldgraf]

I think that @chrisjsewell is seeing if there is appetite within the Sphinx community to collectively suggest to docutils that they use git+github/gitlab, as he thinks that this is an important step to accomplishing @webknjaz's goal of making Sphinx more developer-friendly

[tk0miya]

This is not a Sphinx "community", mere list of issues. I guess extension developers do not check here. It would be better to post this sphinx-devel list instead.

[chrisjsewell]

I will be sure to post on the sphinx mailing lists thanks @tk0miya But, as with sourceforge, I find mailing lists a sub-par platform for discussion over these GitHub issues. Thankfully though we shall soon have GitHub Discussions 😄

[asmeurer]

Where even is docutils-sig? I didn't find it from Google. I understand if you don't like to use issues for discussions, but I hope you can see why this issue is related to Sphinx and this repo @tk0miya. I'm not a fan of mailing lists because there isn't an easy way to subscribe to just one discussion, and also since they are subscribe-only, they tend to much more exclusive in who participates.

[grubert]

good day. i do the releases of docutils ... and try to make some fixes and apply patches

did anyone of you read the former discussions on moving to github ? how is it i have time to talk with everyone on github, while everyone elses time is so scarce they are incapable to learn five commands of subversion ?

all the best :-)

[chrisjsewell]

Dear @grubert, can you point us towards this discussion thanks. As I mention above, I have only found open issues about it with no discussion

[grubert]

there should be something in the docutils developer mailing list archive, shouldnt it ?

https://docutils.sourceforge.io/ -> on the first page it says We welcome all kinds of contributions. There's a To Do list full of interesting ideas awaiting a champion. If you have any questions regarding how to extend Docutils, please feel free to ask on the docutils-develop mailing list.

click docutils-develop three lines down To see the collection of prior postings to the list, visit the Docutils-develop Archives.

click Docutils-develop Archives.

type github in the search box.

somehow obvious or isnt it

[chrisjsewell]

@grubert I have posted on the docutils-develop mailing list, I look forward to your response. I will now move this conversation to there, but for those not subscribed:

---

This post originates from: https://github.com/sphinx-doc/sphinx/issues/8039#issuecomment-668485530, in which I suggested that docutils hosting on Sourceforge is severly hindering its progress and, in turn, affecting aspects of sphinx. Below I recap previous conversation about moving from SourceForge, provide some contextual analysis, and finally ask some questions of the maintainers.

Engelbert gruber suggested I look into the docutils mailing list. As well as the open issue (https://sourceforge.net/p/docutils/feature-requests/58/), here is a summary of past conversation (and extracts):

• 2014: https://sourceforge.net/p/docutils/mailman/docutils-develop/thread/1400868919.8587.120887549.360547E2%40webmail.messagingengine.com/#msg32374278 Post:

  To be a little more blunt: this project looks pretty stagnant, and making it more welcoming to contributors could give it new life. People didn't migrate to git en masse just because because it was new and shiny; it legitimately has positive effects on open source projects.

Answer:

I don't like to go to a host just because "everyone ist doing it". Maybe the hurdle for people > that thrive there would be lower, but at the cost of current developers and contributors. I don't think that the Docutils core will gain much if it attracts a lot of contributors - the > task of keeping the core consistent will become harder.

• 2014: https://sourceforge.net/p/docutils/mailman/docutils-develop/thread/CAEZF91tU2fqeE9YX34pt%2Bc-GZ5y9aNGZ37ytcZ0hiaVL%3Dh04zw%40mail.gmail.com/#msg31874379
  Post: I have a few issues with source forge... I don't find the interface friendly. It's inconsistent. I am not up to date with it, because everyone else is using something else. Even SQLAlchemy moved from bitbucket to github. Here is their rationale: > http://www.sqlalchemy.org/blog/2013/05/25/sqlalchemy-migrated-to-git/

Answer:

subversion does the job, so why change it.

• 2014: https://sourceforge.net/p/docutils/mailman/docutils-develop/thread/CAPE5o16_mH0ptH3HaaW5hQ4kU8y9jzNTF8gwkZLdPyzVgKe6Qw%40mail.gmail.com/#msg32877380 Post: All I can say is that there is room for improvement in the documentation and possibly the code, a need for more work, and I feel encouraging open source participation is about removing every single possible barrier there is to people joining in a helping out

Answer: no reply

• 2018: https://sourceforge.net/p/docutils/mailman/docutils-develop/thread/3c599bf9-626c-be4a-54c4-23fd6d5c9519%40gtec.at/#msg36220506 Post:

  Is there still a plan to move docutils development to github? I think it would ease contributions also for docutils, like for example for "nested inline markup".

Answer: no reply

• 2019: https://sourceforge.net/p/docutils/mailman/docutils-users/thread/CAL1y04q6p7ecChcCfGu2GT0PUjUB6%3DrvL3UAFWm8UUXu9omjeA%40mail.gmail.com/#msg36807539 Post:

  I noticed there are more and more requests in moving the project over to GitHub, may I know why are you not considering moving over?

Answer:

The reasoning is unchanged: don't change a running system unless there is a compelling reason to do so.

• 2020: https://sourceforge.net/p/docutils/mailman/docutils-develop/thread/CABiJP%3DJT9Fw0ngT8WLNENx0SbNDFM%3DxxsCKPJ1q4UV8FX%3DJURw%40mail.gmail.com/#msg36909691 in Please justify your request (make an effort to persuade with evidence). Otherwise the request will be ignored.

You ask for a "compelling reason", here are some contextual statistics:

1. Of 222 packages provided in the https://docs.anaconda.com/anaconda/packages/pkg-docs/ distribution (a good selection of well used packages), only docutils and appscript (which is now defunct) have home pages on sourceforge (see [1] for analysis code): 118 are on github, 2 on bitbucket and 1 on gitlab (most of the rest point to websites like RTD but the source code is still on github)

2. In 2020, sphinx (who have 4 maintainers, the same as docutils) has merged 576 Pull-Request in to their source code, from 84 unique authors: https://github.com/sphinx-doc/sphinx/pulls?q=is%3Apr+is%3Amerged+sort%3Aupdated-desc+closed%3A%3E%3D2020-01-01. In that same time only one patch has been merged into docutils from an external contributor by (well actually one of the sphinx maintainers) https://sourceforge.net/p/docutils/patches/search/?q=status%3Aclosed-wont-fix+or+status%3Aclosed-rejected+or+status%3Aclosed-out-of-date+or+status%3Aclosed-accepted+or+status%3Aclosed+or+status%3Aclosed-fixed

This suggests to me that (a) there is a large community of contributors motivated and able to contribute to python code, specifically in the area of documentation, but (b) none of them are willing to learn a whole new CLI tool (svn or git-svn), workflow (patches over PRs) and UI (sourceforge), since it will not be relevant to any other python package development.

One could argue that docutils is "stable" and does not need any more development. I would beg to differ. (a) in 20 years docutils has still not reached version 1. In fact there has only been 2 releases in three years, both of which are relatively minor: https://docutils.sourceforge.io/RELEASE-NOTES.html (b) there is a full list of TODOs, of which only one item has been removed in three years: https://sourceforge.net/p/docutils/code/8531/log/?path=/trunk/docutils/docs/dev/todo.txt (nested inline markup being a good example of something severely lacking from rST)

There are many many sources online discussing why Github/Gitlab is superior to SourceForge (again this should not be a controversial statement, given that every python package has left it), but here are some extracts:

https://snarky.ca/the-history-behind-the-decision-to-move-python-to-github/

[Offers] a development process that was as simple as possible for core developers to use to review external contributions. This meant things like automated testing for patches, code coverage, patch merging through the browser, etc.

https://news.ycombinator.com/item?id=2739995

Open source software is about programmers collaborating. GitHub makes that incredibly easy and straightforward. Want to join a project? Fork the code and tweak til your hearts content. Your code shows up in the network and other people can find/clone/fork/hack til their hearts are content. Want to contribute to a SF project? Um... contact the dev and hope the project isn't dead?

https://www.quora.com/Why-did-Sourceforge-not-achieve-in-the-2000s-what-GitHub-is-today

GitHub provides a much more streamlined experience for both users and developers of all skill levels. > SourceForge, while a flexible and useful site, just isn't going to get the same kind of adoption based on the the friction and skill level requirements to participate, off-putting ad based model, and lack of simple engagement features.

SO... my questions to the maintainers

1. From the past posts, no judgement, but I get the impression that the main barrier for transition is actually that you don't want any one else to contribute to docutils, and that you would prefer development was only undertaken by yourselves. Is this the case? If so do you feel that you personally have the time available to address these TODOs and bring docutils up to version 1 within a reasonable time period? Or do you expect them never to be completed?
2. If you would like contributions, do you concede that as long as you are on SourceForge, such contributions will be few and far between?

Kind Regards, Chris

---

[1]:

import json, subprocess
packages = json.loads(subprocess.check_output(["pip", "list", "--format", "json"]).decode()) 
homes = {}
for p in packages: 
   print(p["name"])
   data = dict([l.split(":", 1) for l in subprocess.check_output(["pip", "show", p["name"]]).decode().splitlines() if ":" in l])
   homes[p["name"]] = data.get("Home-page", None)

[chrisjsewell]

PS: respect to the sphinx maintainers: 576 PRs merged in 2020 is an awesome feat 🙏 🎉

[grubert]

thanks for the effort

so i try to be as serious as possible

On Tue, 4 Aug 2020 at 21:32, Chris Sewell notifications@github.com wrote:

@grubert https://github.com/grubert I have posted on the docutils-develop mailing list, I look forward to your response. For this not subscribed:

This post originates from: #8039 (comment) https://github.com/sphinx-doc/sphinx/issues/8039#issuecomment-668485530, in which I suggested that docutils hosting on Sourceforge is severly hindering its progress and, in turn, affecting aspects of sphinx. Below I recap previous conversation about moving from SourceForge, provide some contextual analysis, and finally ask some questions of the maintainers.

Engelbert gruber suggested I look into the docutils mailing list. As well as the open issue ( https://sourceforge.net/p/docutils/feature-requests/58/), here is a summary of past conversation (and extracts):

• 2014: https://sourceforge.net/p/docutils/mailman/docutils-develop/thread/1400868919.8587.120887549.360547E2%40webmail.messagingengine.com/#msg32374278 Post: To be a little more blunt: this project looks pretty stagnant, and making it more welcoming to contributors could give it new life. People didn't migrate to git en masse just because because it was new and shiny; it legitimately has positive effects on open source projects. Answer: I don't like to go to a host just because "everyone ist doing it". Maybe the hurdle for people that thrive there would be lower, but at the cost of current developers and contributors. I don't think that the Docutils core will gain much if it attracts a lot of contributors - the task of keeping the core consistent will become harder.

this argument is ignored: keeping the core consistent is a very very strong one

• 2014: https://sourceforge.net/p/docutils/mailman/docutils-develop/thread/CAEZF91tU2fqeE9YX34pt%2Bc-GZ5y9aNGZ37ytcZ0hiaVL%3Dh04zw%40mail.gmail.com/#msg31874379 Post: I have a few issues with source forge... I don't find the interface friendly. It's inconsistent. I am not up to date with it, because everyone else is using something else. Even SQLAlchemy moved from bitbucket to github. Here is their rationale: http://www.sqlalchemy.org/blog/2013/05/25/sqlalchemy-migrated-to-git/ Answer: subversion does the job, so why change it .

and i had some more detailed answer

• install subversion once
• checkout the code once before coding
• update your code
• run the tests
• edit code, test and doc
• check diffs
• commit

what is the impossible thing in this workflow ? beats me absolutely

• 2014: https://sourceforge.net/p/docutils/mailman/docutils-develop/thread/CAPE5o16_mH0ptH3HaaW5hQ4kU8y9jzNTF8gwkZLdPyzVgKe6Qw%40mail.gmail.com/#msg32877380 Post: All I can say is that there is room for improvement in the documentation and possibly the code, a need for more work, and I feel encouraging open source participation is about removing every single possible barrier there is to people joining in a helping out Answer: no reply

who has the work to remove the barriers ... means the current working flow has to changed for everyone in so that others dont have the hurdle of another tool do you understand my argument and it has to be decided by everyone inside

• 2018: https://sourceforge.net/p/docutils/mailman/docutils-develop/thread/3c599bf9-626c-be4a-54c4-23fd6d5c9519%40gtec.at/#msg36220506 Post: Is there still a plan to move docutils development to github? I think it would ease contributions also for docutils, like for example for "nested inline markup". Answer: no reply

• 2019: https://sourceforge.net/p/docutils/mailman/docutils-users/thread/CAL1y04q6p7ecChcCfGu2GT0PUjUB6%3DrvL3UAFWm8UUXu9omjeA%40mail.gmail.com/#msg36807539 Post: I noticed there are more and more requests in moving the project over to GitHub, may I know why are you not considering moving over? Answer: The reasoning is unchanged: don't change a running system unless there is a compelling reason to do so.

• 2020: https://sourceforge.net/p/docutils/mailman/docutils-develop/thread/CABiJP%3DJT9Fw0ngT8WLNENx0SbNDFM%3DxxsCKPJ1q4UV8FX%3DJURw%40mail.gmail.com/#msg36909691 in Please justify your request (make an effort to persuade with evidence). Otherwise the request will be ignored.

You ask for a "compelling reason", here are some contextual statistics:

1.

Of 222 packages provided in the https://docs.anaconda.com/anaconda/packages/pkg-docs/ distribution (a good selection of well used packages), only docutils and appscript (which is now defunct) have home pages on sourceforge (see [1] for analysis code): 118 are on github, 2 on bitbucket and 1 on gitlab (most of the rest point to websites like RTD but the source code is still on github) 2.

In 2020, sphinx (who have 4 maintainers, the same as docutils) has merged 576 Pull-Request in to their source code, from 84 unique authors: https://github.com/sphinx-doc/sphinx/pulls?q=is%3Apr+is%3Amerged+sort%3Aupdated-desc+closed%3A%3E%3D2020-01-01 . In that same time only one patch has been merged into docutils from an external contributor by (well actually one of the sphinx maintainers) https://sourceforge.net/p/docutils/patches/search/?q=status%3Aclosed-wont-fix+or+status%3Aclosed-rejected+or+status%3Aclosed-out-of-date+or+status%3Aclosed-accepted+or+status%3Aclosed+or+status%3Aclosed-fixed

This suggests to me that (a) there is a large community of contributors motivated and able to contribute to python code, specifically in the area of documentation, but (b) none of them are willing to learn a whole new CLI tool (svn or git-svn), workflow (patches over PRs) and UI (sourceforge, since it will not be relevant to any other python package development.

i spend more time reading your mail, than it takes to learn svn that means i am willing to work and contribute

One could argue that docutils is "stable" and does not need any more development. I would beg to differ. (a) in 20 years docutils has still not reached version 1. In fact there has only been 2 releases in three years, both of which are relatively minor: https://docutils.sourceforge.io/RELEASE-NOTES.html

most projects have rolling releases docutils readme (AKAIR) says "the repository is to be considered more stable and mature" since 0.1 i think this is a rolling release since day one .

(b) there is a full list of TODOs, of which only one item has been removed in three years: https://sourceforge.net/p/docutils/code/8531/log/?path=/trunk/docutils/docs/dev/todo.txt (nested inline markup being a good example of something severely lacking from rST)

There are many many sources online discussing why Github/Gitlab is superior to SourForge (again this should not be a controversial statement, given that every python package has left it), but here are some extracts:

three command update, diff and commit for svn git needs more ... what is better in needs more ? i am waiting for the answer for years now

https://snarky.ca/the-history-behind-the-decision-to-move-python-to-github/ [Offers] a development process that was as simple as possible for core developers to use to review external contributions. This meant things like automated testing for patches, code coverage, patch merging through the browser, etc. https://news.ycombinator.com/item?id=2739995 Open source software is about programmers collaborating. GitHub makes that incredibly easy and straightforward. Want to join a project? Fork the code and tweak til your hearts content. Your code shows up in the network and other people can find/clone/fork/hack til their hearts are content. Want to contribute to a SF project? Um... contact the dev and hope the project isn't dead?

https://www.quora.com/Why-did-Sourceforge-not-achieve-in-the-2000s-what-GitHub-is-today GitHub provides a much more streamlined experience for both users and developers of all skill levels. SourceForge, while a flexible and useful site, just isn't going to get the same kind of adoption based on the the friction and skill level requirements to participate, off-putting ad based model, and lack of simple engagement features.

SO... my questions to the maintainers

1. From the past posts, no judgement, but I get the impression that the main barrier for transition is actually that you don't want any one else to contribute to docutils, and that you would prefer development was only undertaken by yourselves.

because no one was rejected the developer access i think i could take this as offensive

1. Is this the case? If so do you feel that you personally have the time time to avilable to adress these TODOs and bring docutils up to version 1 within a resoinable time period? Or do you expect them never to be completed?

docutils is in production use so what do you miss for your version 1 ... and if you follow discussions you notice that david is absolutely picky about not breaking the code therefore i assume the TODO list will be implemented as needed not as a feat.

1. If you would like contributions, do you concede that as long as you are on SourceForge, such contributions will be few and far between?

git patch is impossible is it ?

i am not against github (maybe i would prefer gitlab) but this demands a change for everyone already in the project so i can not decide and if it is decided moving to github is like a release it has to be done very thoughtfully means takes a lot of hours

cheers

Kind Regards, Chris

[1]: import json, subprocess packages = json.loads(subprocess.check_output(["pip", "list", "--format", "json"]).decode()) homes = {} for p in packages: print(p["name"]) data = dict([l.split(":", 1) for l in subprocess.check_output(["pip", "show", p["name"]]).decode().splitlines() if ":" in l]) homes[p["name"]] = data.get("Home-page", None)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/sphinx-doc/sphinx/issues/8039#issuecomment-668784436, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAHEYA3ZDKGAXACIGTV6MWLR7BO6BANCNFSM4PSSE4PQ .

[chrisjsewell]

@grubert out of respect to the sphinx devs and @webknjaz 's original intent for this issue I don't wish to draw this discussion out here. Can you please respond on your own mailing list and I will reply there

[webknjaz]

I don't think the developers have ever considered the use of Subversion rather than git to be a serious objection rather than a statement of tastes, especially given the availability of git-svn. The superiority of git for such a small project is certainly contestable. (E.g., svnvsgit.com)

FWIW that site was updated 4 years ago and has outdated statements like telling people that the stats is 47% svn and 38% while the link it offers as a proof says 24% svn and 71% git. Besides, it's not really about the size of the project but about the impact it has...

re: offering patches, I think one of the main topics of this thread is "there are people who would love to contribute patches but doing so via subversion and sourceforge makes it much harder to do so, generally hard enough that it just doesn't happen".

This is totally true but I also want to add that there's no native email-based workflow in svn forcing folks figure out what's going on in those non-straightforward UIs: it's not code-oriented so the first place you click is "Files" and then you don't see any code and get confused. AFAICS there's even no CI so there's no transparency into what's going on, no review tooling that is present in literally all other solutions — whether it's GH's PRs or Gerrit in other places — there's always review possibilities for adding context by making discussions inline, in places they refer to.

P.S. I've added some comments wrt other code hosting solutions @ https://sourceforge.net/p/docutils/feature-requests/58/#be15

Makes it harder ... how? Isn't it really more than a matter of taste? (I'm open to the possibility that there is a real issue here, but I rather doubt it; see the link I included.) Why is git-svn not good enough for those who find Subversion annoying?

It's probably useable as a band-aid but still doesn't solve the tooling and ecosystem problems above...

---

Alright, let's get back to the original questions.

I find it very positive that this discussion is happening. I'm wondering what the way forward is:

• Documenting things ?
• Creating a library on top of docutils that provides a sane abstraction layer ?
• Creating a new library replacing docutils for the needs of Sphinx ? (in both case, the compatibility requirements will be very complex)

In short, what do you imagine needs to be done so that, in a plausible future not so far away, people don't have to make a ritual sacrifice in order to make new sphinx (/RST) features ? This could be as "simple" as an extensive documentation project within docutils with a lot of examples, code snippets, and written for people who don't already know the lib.

I'd say yes to all 3 but the short term would definitely be writing the docs. Then, making better APIs/abstractions. And finally, considering a better lib or a fork or something but this stage would need to be driven by somebody resourceful and sphinx devs would need to agree to this too, I guess.

FWIW the first two steps are inarguably a good start.

[grubert]

On Fri, 7 Aug 2020 at 00:09, Sviatoslav Sydorenko notifications@github.com wrote: BIG CUT to shorten this

---

Alright, let's get back to the original questions.

I find it very positive that this discussion is happening. I'm wondering what the way forward is:

• Documenting things ?

• Creating a library on top of docutils that provides a sane abstraction layer ?

the reason why david argues against docutils version 1.0 is because he does not consider the API as stable putting another wrapper around could be

• con: why not change it in docutils. take the on hand use case to better define the api
• pro: make a API-subset for this use case

the reason why changes in docutils are not easy is because we do not want to break other peoples code/patches if not used via the supplied forntends changes are to be taken cautious.

so documentation and feature requests, samples (the multitasking frontend for example) are very welcome any contribution is considered

• Creating a new library replacing docutils for the needs of Sphinx ? (in both case, the compatibility requirements will be very complex)

In short, what do you imagine needs to be done so that, in a plausible future not so far away, people don't have to make a ritual sacrifice in order to make new sphinx (/RST) features ? This could be as "simple" as an extensive documentation project within docutils with a lot of examples, code snippets, and written for people who don't already know the lib.

I'd say yes to all 3 but the short term would definitely be writing the docs. Then, making better APIs/abstractions. And finally, considering a better lib or a fork or something but this stage would need to be driven by somebody resourceful and sphinx devs would need to agree to this too, I guess.

FWIW the first two steps are inarguably a good start.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/sphinx-doc/sphinx/issues/8039#issuecomment-670215878, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAHEYA333HVNWX2RFGGMIQ3R7MSYPANCNFSM4PSSE4PQ .

[chrisjsewell]

Note I have now created: https://github.com/chrisjsewell/docutils As it states in the README, this package shows how a migration can be done in a very simple, autonomous manner:

• migrating code, commit history (with authors) and branches using the GitHub Importer (this takes about 10 minutes),
• migrating tickets to issues using a python script that interfaces with the SourceForge and GitHub REST APIs (this is both autonomous and idempotent and takes about 10 minutes).

The main branch derives from the svn trunk, then here on the develop branch this README has been added and a GitHub workflow for Continuous Integration testing and coverage.

I think this covers the majority of a migration? (if not it can probably also be done via REST)

---

Unfortunately @grubert did not reply to me on the mailing list, so I will make some responses here:

what is the impossible thing in this workflow ?

As I have already alluded to, an argument over how easy/hard the workflow is, is essentially irrelevant. The fact is that no one is contributing, which shows that it is a barrier that few are willing to hurdle. That being said, I'd note that you don't also include the steps that come after a commit: creating a patch, signing up to sourceforge, submitting the patch, having to go through a sub-optimal (to PRs) review process. Authorial attribution is also intrinsically poor, since all commits are assigned the author of the maintainer, as opposed to the person who submitted the patch.

I think i could take this as offensive

I apologise if you felt this way, this is not my intent. But also I want to be very clear in my opinion that this inaction is driving away potential contributors.

the reason why david argues against docutils version 1.0 is because he does not consider the API as stable

the reason why changes in docutils are not easy is because we do not want to break other peoples code/patches

This is what I struggle to understand, you are saying that the docutils API is both unstable and yet will very rarely (if ever) be changed. These statements seem contradictory to me? Also, isn't that the point of semantic versioning; that you can introduce breaking changes between major versions, and people just pin their dependencies to a major version until they are ready to upgrade?

if it is decided moving to github is like a release it has to be done very thoughtfully means takes a lot of hours

But there are people out there, like myself, that would be willing to help with this 😄

[asmeurer]

On the off chance this actually happens, I should note that there are better ways to migrate issues that don't result in every comment author being the person who does the migration.

[chrisjsewell]

How do you do that?

[asmeurer]

When we did it with SymPy, we had to contact GitHub support and they helped us do it. I don't know if it can be done nowadays with the API. If not, then hopefully GitHub support is still willing to help out.

[ewjoachim]

Hey :)

While the works on this is is interesting, the Sphinx maintainer has stated multiple times that this was not the place to have it, and I'm feeling it very awkard that, yet, we continue this discussion here. Would you mind taking the discussion elsewhere (e.g. on the new repo you created) and closing this issue ?

Thanks.

[chrisjsewell]

FYI, maybe more applicable to a whole sphinx extension than just writing a directive, but I think the process outline here is quite helpful (since I wrote it 😆): https://www.sphinx-doc.org/en/master/extdev/appapi.html?highlight=config-inited#sphinx-core-events

Also in https://github.com/executablebooks/sphinx-ext-autodoc I started to play around with "auto-documenting" a sphinx environment (given a certain conf.py), which I think could also be useful to assess how a sphinx extension interacts with the sphinx build.

[brechtm]

Sorry to bump this off-topic issue, but my intention is to move this discussion elsewhere (see below).

As the author of two projects (rinohtype being one of them) that are strongly tied to docutils, I am not too happy about the current state of docutils and the reStructuredText ecosystem as a whole. While I am very happy with the capabilities reStructuredText (though docutils and Sphinx) offers, we should not ignore that there are some major issues that are holding back its potential. My two main gripes are:

1. The official homepage for reStructuredText is a set of pages on the docutils project's website. The look and organization of this website is simply not up to standards. reStructuredText could be a lot more popular if we had a proper homepage that is inviting to newcomers and makes it easier for anyone to find the information they are looking for.

2. Building on top of docutils, and Sphinx by extension, is much harder than it should be. I have fantasized about writing a reStructuredText parser from scratch, but I would be fooling myself to think that I can produce anything better than docutils in a reasonable timeframe. Instead, it is better to try and improve the existing tools or build a friendly API on top of them. However, this is made rather difficult by the docutils team sticking to Subversion and SourceForge. While I did submit a couple of patches in the past, having to deal with Subversion in 2020 (for one) does create an obstacle to contributing.

I don't think we can ask the docutils team to fix these issues for us. Instead, let's find out whether there are enough of us to make these things happen. I invite anyone interested to join the discussion here: reStructuredText/startup#1.

Tagging a couple of people that might be interested: @lextm @ericholscher @jdillard @KubaO. And some who wrote reStructuredText documentation/tutorials: @pydanny @cokelaer.

[tk0miya]

It seems this discussion was moved to https://github.com/reStructuredText/startup/issues/1. So I'm closing this. Please let me know if there is something I can do to improve docutils.

[AA-Turner]

For posterity, new parsing functions were added to SphinxDirective in #12492.

A