Convert `dropdown` directive without using `eval-rst`

[mbercx]

Is your feature request related to a problem? Please describe.

When trying to convert a rubric directive, e.g.:

.. rubric:: Footnotes

.. [#f1] We purposefully do not provide advanced commands for crystal structure manipulation in AiiDA, because python packages that accomplish such tasks already exist (such as ASE or pymatgen).

rst2myst parse -f produces:

:::{rubric} Footnotes

:::

[^f1]: We purposefully do not provide advanced commands for crystal structure manipulation in AiiDA, because python packages that accomplish such tasks already exist (such as ASE or pymatgen).

This causes the build to fail with the following warning:

WARNING: toctree contains reference to document 'sections/fundamentals/basics' that doesn't have a title: no link will be generated

Proposed solution

I tried using the extended markdown syntax for footnotes, but maybe mystnb doesn't support this? If no support can be added, maybe a warning would be useful during parsing, since the warning returned during the build does not point towards the footnotes issue.

[welcome[bot]]

Thanks for opening your first issue here! Engagement like this is essential for open source projects! :hugs:
If you haven't done so already, check out EBP's Code of Conduct. Also, please try to follow the issue template as it helps other community members to contribute more effectively.
If your issue is a feature request, others may react to it, to raise its prominence (see Feature Voting).
Welcome to the EBP community! :tada:

[chrisjsewell]

Heyo, footnotes are indeed supported. But this does not seem to be the root of your issue. As the warning say, it cannot find a header in sections/fundamentals/basics. What does the top of the file look like? Does it have a header, i.e.

# My Header

[mbercx]

Indeed, it does. But it seems I passed the wrong warning, this one seems more critical:

/Users/mbercx/python/code/aiida-tutorials/docs/sections/fundamentals/basics.md:1: WARNING: MyST-NB: Conversion to notebook failed: list index out of range

Maybe this is related to the fact that ..dropdown directives seem to be wrapped in eval-rst:

```{eval-rst}
.. dropdown:: Defining a structure and storing it in the database
...

Indeed, moving the offending footnote out of the eval-rst fixes the issue as well. I also tried putting the footnote elsewhere before conversion and this worked, however, only if I leave 1 line empty after the footnote. Two lines or more leads in the index out of range error above... 🤔 I swear, I'm not going insane. 😅

@chrisjsewell maybe try the conversion with the basics.rst file and have a look. You can find it here:

https://filedn.com/lsOzB8TTUIDz2WkFj8o6qhp/basics.rst

[chrisjsewell]

Ok cheers will do, last question, what version of myst-nb, myst-parser and markdown-it-py are you using (well if you just want to paste the output of pip freeze)

[chrisjsewell]

It's quite possibly because you are using a version before this bug fix: https://github.com/executablebooks/markdown-it-py/commit/a40f6a80f84f2c02e999c6b77f8112c050f8cbe6

I'm basically going to release a myst-nb with this in imminently (you can even try myst-nb==0.13.0a1)

[mbercx]

Here's the pip freeze:

pip freeze
alabaster==0.7.12
anyio==3.0.1
appdirs==1.4.4
appnope==0.1.2
argon2-cffi==20.1.0
async-generator==1.10
attrs==20.3.0
Babel==2.9.1
backcall==0.2.0
beautifulsoup4==4.9.3
black==21.5b1
bleach==3.3.0
certifi==2020.12.5
cffi==1.14.5
cfgv==3.2.0
chardet==4.0.0
click==7.1.2
colorama==0.4.4
decorator==5.0.7
defusedxml==0.7.1
distlib==0.3.1
docutils==0.15
entrypoints==0.3
filelock==3.0.12
gitdb==4.0.7
GitPython==3.1.17
identify==2.2.4
idna==2.10
imagesize==1.2.0
importlib-metadata==4.0.1
importlib-resources==3.3.1
ipykernel==5.5.5
ipython==7.23.1
ipython-genutils==0.2.0
ipywidgets==7.6.3
jedi==0.18.0
Jinja2==3.0.0
jsonschema==3.2.0
jupyter-cache==0.4.2
jupyter-client==6.1.12
jupyter-core==4.7.1
jupyter-server==1.7.0
jupyter-server-mathjax==0.2.2
jupyter-sphinx==0.3.2
jupyterlab-widgets==1.0.0
markdown-it-py==0.6.2
MarkupSafe==2.0.0
matplotlib-inline==0.1.2
mdit-py-plugins==0.2.6
mistune==0.8.4
mypy-extensions==0.4.3
myst-nb==0.12.3
myst-parser==0.13.7
nbclient==0.5.3
nbconvert==5.6.1
nbdime==3.0.0
nbformat==5.1.3
nest-asyncio==1.5.1
nodeenv==1.6.0
notebook==6.3.0
packaging==20.9
pandocfilters==1.4.3
parso==0.8.2
pathspec==0.8.1
pexpect==4.8.0
pickleshare==0.7.5
pre-commit==2.12.1
prometheus-client==0.10.1
prompt-toolkit==3.0.18
ptyprocess==0.7.0
pycparser==2.20
pydata-sphinx-theme==0.6.3
Pygments==2.9.0
pyparsing==2.4.7
pyrsistent==0.17.3
python-dateutil==2.8.1
pytz==2021.1
PyYAML==5.4.1
pyzmq==22.0.3
regex==2021.4.4
requests==2.25.1
rst-to-myst==0.1.2
Send2Trash==1.5.0
six==1.16.0
smmap==4.0.0
sniffio==1.2.0
snowballstemmer==2.1.0
soupsieve==2.2.1
Sphinx==3.5.4
sphinx-book-theme==0.1.0
sphinx-copybutton==0.3.1
sphinx-panels==0.5.2
sphinx-togglebutton==0.2.3
sphinxcontrib-applehelp==1.0.2
sphinxcontrib-devhelp==1.0.2
sphinxcontrib-htmlhelp==1.0.3
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.3
sphinxcontrib-serializinghtml==1.1.4
SQLAlchemy==1.3.24
terminado==0.9.5
testpath==0.4.4
toml==0.10.2
tornado==6.1
traitlets==5.0.5
urllib3==1.26.4
virtualenv==20.4.6
wcwidth==0.2.5
webencodings==0.5.1
websocket-client==0.59.0
widgetsnbextension==3.5.1
zipp==3.4.1

Will try the pre-release, thanks! 👍

[mbercx]

I'm basically going to release a myst-nb with this in imminently (you can even try myst-nb==0.13.0a1)

With the prerelease I no longer run into the "index out of range" issue, and the document is built. 👍 The number of empty lines at the end of the file also no longer have any influence (phew 😅 ). So when the footnote is outside the dropdown, it works perfectly. 👌

When the footnote is inside the dropdown, it doesn't get converted (being in the rst-eval) and I get the following warning:

/Users/mbercx/python/code/aiida-tutorials/docs/sections/fundamentals/basics.md:1622: WARNING: Footnote [#] is not referenced.

[mbercx]

So basically the issue boils down to the fact that:

.. dropdown:: Dropdown

    Hidden content

Gets converted into:

```{eval-rst}
.. dropdown:: Dropdown

    Hidden content
```#

And not

```{dropdown} Dropdown
Hidden content
```#

I suppose there is some reason for this, but this causes problems with the footnotes, since it seems the markdown footnotes can't be referenced from within the eval-rst directive.

[chrisjsewell]

Note, if you read the advanced instructions, you can change how dropdown is converted with the -c/--conversions option