Include directive: Allow separate includes for reference links and reference definition

[eyllanesc]

Describe the bug

foo.md

<!-- contents-begin-md -->

Hello world [Test][test].

<!-- contents-end-md -->

MyST-Parser is awesome!!!

<!-- references-begin-md -->

[test]: http://example.com/

<!-- references-end-md -->

bar.rst

.. contents-begin-rst

Hello world `Test <test_>`__

.. contents-end-rst

.. references-begin-rst

.. _test: http://example.com/

.. references-end-rst

1.

When I include parts of foo.md in another md.

```{include} foo.md
---
start-after: <!-- contents-begin-md -->
end-before: <!-- contents-end-md -->
---

---
start-after: <!-- references-begin-md -->
end-before: <!-- references-end-md -->
---

When rendering you get where the reference is lost: 

```raw
Hello world [Test][test].

2.

When I include parts of bar.rst in an md.

```{eval-rst}
.. include:: bar.rst
   :start-after: contents-begin-rst
   :end-before: contents-end-rst

.. include:: bar.rst
   :start-after: references-begin-rst
   :end-before: references-end-rst

So it renders as a link but points to: `host/#test` instead of to `http://example.com/`

# 3.

If I include  foo.md in an .rst: 

```rst
.. include:: foo.md
   :parser: myst_parser.sphinx_
   :start-after: <!-- contents-begin-md -->
   :end-before: <!-- contents-end-md -->

.. include:: foo.md
   :parser: myst_parser.sphinx_
   :start-after: <!-- references-begin-md -->
   :end-before: <!-- references-end-md -->

Hello world [Test][test]. is obtained. The link is not rendered.

4.

When I include parts of bar.rst in another .rst

.. include:: bar.rst
   :start-after: contents-begin-rst
   :end-before: contents-end-rst

.. include:: bar.rst
   :start-after: references-begin-rst
   :end-before: references-end-rst

It renders correctly.

Reproduce the bug

Source code: https://github.com/eyllanesc/myst_bug Output: https://myst-bug.readthedocs.io/en/latest/index.html

List your environment

• sphinx==4.4.0
• myst_parser==0.16.1, 0.17.0

[welcome[bot]]

Thanks for opening your first issue here! Engagement like this is essential for open source projects! :hugs:
If you haven't done so already, check out EBP's Code of Conduct. Also, please try to follow the issue template as it helps other community members to contribute more effectively.
If your issue is a feature request, others may react to it, to raise its prominence (see Feature Voting).
Welcome to the EBP community! :tada:

[thht]

I have the exact same problem. Any idea, how to solve this?

[kinow]

I think I don't have the exact issue, but this this part about reference links not being rendered:

Hello world [Test][test]. is obtained. The link is not rendered.

In my case, I have an existing documentation that was ported from Jekyll and had a links.md with all the reference links definitions. Including that page is giving me the same behaviour, of links not rendered.

Trying to understand now whether it's related to this issue, or if I better to create a separate issue :+1:

[thht]

Ok, I did some digging.

The problem is basically the order in which things are done:

1. Convert the markdown in a file to a list of tokens. References are already resolved here and references defined within the same file are resolved correctly.
2. Evaluate all the directives. This includes the {include} directive.

This means that if the included file defines some references, they are not available, when the references in the original file are resolved.

There would be a few ways to potentially solve this:

1. Have MystParser.parse intercept any include directives before calling render and just add that content to the inputstring. You would need to make sure to handle recursion correctly.
2. Write a markdown_it plugin that provides the include file functionality. I think, this might be doable but at the moment, myst-parser comes with a set of pre-activated plugins and I could not find a way to tell it to activate more...
3. Anything else?

For me, having a collection of references in an included file is really something that makes my life easier...

@chrisjsewell and maybe @choldgraf : is there any chance of this getting included? I would be happy and capable of contributing code and time. But I would need some feedback from you:

1. Would you be willing to include this feature? If yes, under what conditions (except for the obvious like code quality, flake, unit tests)?
2. What strategy / approach would you prefer?

[mawieland]

I might have found a "workaround" for the first problem . Using the MySt directive with syntax from rst works in my case:

(note the extra colon before "start-after" and missing dashes)

```{include} ./my_file.md
:start-after: dummy-label start
:end-before: dummy-label end

myst-parser                0.18.0
Sphinx                        5.1.1  

[chrisjsewell]

This is a core limitation of commonmark I'm afraid; reference definitions have to be parsed before reference links:: https://github.com/commonmark/commonmark-spec/issues/702 (this behaviour is different to rST

So it's non-trivial to change this behaviour for includes