Support for restructuredText?

[jboxman]

I've recently been using restructuredText for building documentation sets. I've used Rewrap successfully in the past with Markdown and grown quite fond of it. It doesn't seem to understand .rst though. Would it be feasible to add support for it?

[stkb]

For sure 👍

[epaul13]

Yes, something really cool to have for bullet-point lists in rst. For now, I need to set the right indent on the second line of a bullet and use rewrap from this line to get the correct behavior :) Great work anyway.

[183amir]

Coming over from sublime text, I was used to https://github.com/ehuss/Sublime-Wrap-Plus which understood rst files and Python docstrings. Hopefully, we can have this feature here as well.

[jessicah]

Just started using this, and the biggest issue I'm running into with restructuredText is that the heading markers get moved to the end of the headings when rewrapping the whole document:

Some Heading
============

becomes

Some Heading ============

The wrapping as writing works pretty good though :)

[jessicah]

Kind of seems like plainText could do with the check from Markdown: https://github.com/stkb/Rewrap/blob/master/core/Parsing.Markdown.fs#L128

[stkb]

ReStructuredText support will be in the next version. It's not quite finished but has been quite an undertaking so I want to make available the progress so far.

• [x] Paragraphs
• [x] Transitions
• [x] Section titles
• [x] Block quotes
• [x] Bullet lists
• [x] Enumerated Lists
  • [ ] Order validation (eg a valid list must have items 1, 2, 3...) Not done yet; maybe won't be
• [x] Definition lists
  • [ ] With classifiers: not checked yet
• [x] Field lists (should be fine in most cases, but not completely accurate yet)
• [ ] Option lists (not supported yet; wrapping will mess up)
• [x] Literal Blocks
• [x] Line Blocks
• [x] DocTest blocks
• [ ] Tables (will not be wrapped)
  • [x] Grid Tables
  • [ ] Simple Tables (not supported yet; wrapping will mess up)
• [ ] Explicit markup blocks
  • [x] Footnotes & citations (same as field lists, fine in most cases but not completely accurate)
  • [ ] Hyperlink targets, directives, substitution definitions, comments: Detected but not wrapped

Works in standalone .rst files or in Python docstrings. I'm assuming that # ... line comments should stay as markdown but correct me if I'm wrong. Eventually I want to make that configurable though that will be another release away. Also I've kept Julia as markdown for now as from what I've read it's more commonly used than RST there.

There may well be some bugs still but you can test it on v1.16.0-beta.3

[jboxman]

Glad to hear it! Won't be able to test as I've moved on from restructuredText some years ago now.

[stkb]

@jboxman No worries, it has indeed been a few years :) So for anyone else still interested.

[timblakely]

Thanks for the extension @stkb ! Updated to pre-release v17.3.0 and am still seeing the python docstring formatting errors from #89.

Initial state:

#                                                                     80c ---> |
def restore_model(checkpoint_dir: Optional[str] = None,
                  saved_model: Optional[str] = None) -> TFModel:
  """Restore a trained model and return a prediction function.

  Args:
    checkpoint_dir: Path to the checkpoint directory containing one or more checkpoints. Defaults to None. 
    saved_model: Path to the saved model to apply. Model expected to have a function that takes `input_data` of shape [B, H, W, 1] and parameter `clip_and_round=True`. Defaults to None.

  Raises:
    ValueError: If no checkpoint could be loaded.

  Returns:
    Loaded TF model.
  """
  # ...

Current behavior:

#                                                                     80c ---> |
def restore_model(checkpoint_dir: Optional[str] = None,
                  saved_model: Optional[str] = None) -> TFModel:
  """Restore a trained model and return a prediction function.

  Args:
    checkpoint_dir: Path to the checkpoint directory containing one or more
    checkpoints. Defaults to None. saved_model: Path to the saved model to
    apply. Model expected to have a function that takes `input_data` of shape
    [B, H, W, 1] and parameter `clip_and_round=True`. Defaults to None.

  Raises:
      ValueError: If no checkpoint could be loaded.

  Returns:
      Loaded TF model.
  """
  # ...

Expected behavior:

#                                                                     80c ---> |
def restore_model(checkpoint_dir: Optional[str] = None,
                  saved_model: Optional[str] = None) -> TFModel:
  """Restore a trained model and return a prediction function.

  Args:
    checkpoint_dir: Path to the checkpoint directory containing one or more
      checkpoints. Defaults to None.
    saved_model: Path to the saved model to apply. Model expected to have a
      function that takes `input_data` of shape [B, H, W, 1] and parameter
      `clip_and_round=True`. Defaults to None.

  Raises:
    ValueError: If no checkpoint could be loaded.

  Returns:
    Loaded TF model.
  """
  # ...

I can't seem to tell if I'm triggering the restructuredText style (or even if this is the expected wrapping behavior when using restructuredText :)

[stkb]

Hi @timblakely,

I tried your example myself in a .py file and yes it does seem to be using the restructuredText implementation.

Firstly the Raises/Return sections: In your given output the contents get indented 2 extra spaces:

#                                                                     80c ---> |
  """ [...]

  Raises:
      ValueError: If no checkpoint could be loaded.

  Returns:
      Loaded TF model.
  """

I don't know if that was just a typo or you're really seeing that. For me they stay the same indent, as expected:

#                                                                     80c ---> |
  """ [...]

  Raises:
    ValueError: If no checkpoint could be loaded.

  Returns:
    Loaded TF model.
  """
  # ...

Otherwise it may be a tabs/spaces issue (are you using tabs?)

---

Now I think the meat of the issue is the Args: section and that Rewrap is treating everything there as a single paragraph, whereas you're expecting the checkpoint_dir and saved_model paragraphs to remain on separate lines?

I don't use RST myself so I've only gone by the spec and what this online editor does. I don't see it in the spec that those should be separate paragraphs and if I input the source text in the editor then it counts everything under Args: as a single paragraph.

If I add colons to the front of those arg names, then it does keep them separate (it becomes a field list). Output after wrapping:

#                                                                     80c ---> |
def restore_model(checkpoint_dir: Optional[str] = None,
                  saved_model: Optional[str] = None) -> TFModel:
  """Restore a trained model and return a prediction function.

  Args:
    :checkpoint_dir: Path to the checkpoint directory containing one or more
      checkpoints. Defaults to None. 
    :saved_model: Path to the saved model to apply. Model expected to have a
      function that takes `input_data` of shape [B, H, W, 1] and parameter
      `clip_and_round=True`. Defaults to None.

  Raises:
    ValueError: If no checkpoint could be loaded.

  Returns:
    Loaded TF model.
  """
  # ...

As I say I don't use RST myself so if there's a 'flavor' or tool you're using that does treat any line that starts with + ":" as a new paragraph then let me know.

[timblakely]

I don't know if that was just a typo or you're really seeing that. For me they stay the same indent, as expected:

Nope, that was a bone-headed move on my part: I realized halfway through my report that I actually had the spacing set to four instead of two in my pydoc template, so corrected it and continued on with the bug report... without fixing the first part of it. Apologies, please ignore those leading whitespace changes! :weary:

Now I think the meat of the issue is the Args: section and that Rewrap is treating everything there as a single paragraph, whereas you're expecting the checkpoint_dir and saved_model paragraphs to remain on separate lines?

Bingo! This follows the Google (No Types) format of python docstring, so hoping to be able to wrap like that.

I don't use RST myself

I'll be completely honest and say that I have no idea what RST is myself, but was optimistic when I saw a similar mention over in #89 🙃 Was hoping that that would automagically fix formatting, but unfortunately not the case it seems.

If I add colons to the front of those arg names, then it does keep them separate (it becomes a field list). Output after wrapping:

Interesting. That's basically exactly the behavior I'd love to see, minus the having-to-prepend-a-:part. As a workaround I guess I could create all docstrings from a template that pre-populates the :, then just do a file-wide regex sub along the lines of ^(\s+): > $1 to get rid of all the leading colons. I'd love to have this capability built into Rerwrap natively, but I admit I know little of the internal working of the extension and don't know how hard this would be to shoehorn into the existing parsing 🥴

[stkb]

I wanted to have something else implemented before replying but haven't had time yet.

Firstly I do plan on adding a feature where you can add your own find-and-replace operations that take place when wrapping.

Yeah the problem is the Google documentation format you're using isn't RST, so it won't behave the same. I do also plan on letting the user choose which parser is used for comments, be it markdown, RST, plaintext or whatever.

If you feel up for writing a spec for the Google format (see this comment on writing specs) I can implement it. The most information I can find on the format is here. However it looks like you won't to just rewrap the whole comment without messing things up; in the example on that page, there seems no way for it to tell that that code block is a non-wrappable code block and not just any other wrappable paragraph.