Closed Laupetin closed 1 month ago
Welcome to AsyncAPI. Thanks a lot for reporting your first issue. Please check out our contributors guide and the instructions about a basic recommended setup useful for opening a pull request.
Keep in mind there are also other channels you can use to interact with AsyncAPI community. For more details check out this issue.
@Laupetin Please share files, structured the way you mentioned as an archive, a comment, or a separate branch in https://github.com/Laupetin/asyncapi-bundler Anonymized, but as close to real as possible (or use one of the examples to keep the structure completely generic,) before we move further with this issue.
@aeworxet I created an example for demonstration purposes, see https://github.com/asyncapi/bundler/pull/179#issuecomment-2242420012
I have requested additional opinions on this subject in https://asyncapi.slack.com/archives/CQVJXFNQL/p1721883187801609
The exerpt:
#
While the default behavior of Bundler
cannot be changed because it will violate the JSON standard's behavior
Resolution is performed relative to the referring document.
around which the Bundler
is built, I still think that this usecase is bigger than just one user's request, because this directory structure seems useful for documenting complex APIs.
So, based on this, I lean toward adding something like the --traverse-subdirs
(name is free for proposals) option, which:
if specified
--base
simultaneously specified, will traverse subdirectories, resolve $ref
s of each YAML it finds there, and merge each resulting bundled sub-YAML into the main - --base
- YAML.--base
NOT specified, will consider --base
the 'usual' YAML (that was provided as an argument to the Bundler
) and still do everything described aboveif NOT specified
Bundler
will retain its current behavior, considering the flat $ref
structure as it does now (dereferencing only explicit references it is able to find, relative to the referring document)Does the addition of such an option make sense? Should it be doing what I described? Do you have other thoughts on this subject?
#
if NOT specified
Bundler
will retain its current behavior, considering the flat$ref
structure as it does now (dereferencing only explicit references it is able to find, relative to the referring document)
I think there is a misunderstanding there :sweat_smile: The issue is that the bundler does not do that currently. What I am suggesting is making him do that.
For example if your project and working directory is located in /home/example
and you try to bundle docs/asyncapi/index.yaml
and inside it refers to ./schema.json
then:
/home/example/schema.json
which is not what the linked standard suggests./home/example/docs/asyncapi/schema.json
(which is relative to the document) and that is also what other asyncapi tools like the generator and validator doI'm sorry if me talking about about glob in the PR made things more confusing :sweat_smile: This issue was just meant for trying to change the behaviour to be resolved to "relative to the document".
Before I didn't know it was defined in the standard. Since it is I guess this could also be considered a bug then instead of a "feature"?
It is still not fully understandable whether you expect Bundler
to do what it was initially made for, because there are no $ref
s in the ./docs/asyncapi/index.yaml
at all.
Please read https://asyncapi.slack.com/archives/CQVJXFNQL/p1716582167100109 and suggest whether it is the Bundler
's documentation that should be made clearer, or whether a user of Bundler
intuitively expects something else?
It is still not fully understandable whether you expect
Bundler
to do what it was initially made for, because there are no$ref
s in the./docs/asyncapi/index.yaml
at all.
The index.yaml
does not but the docs/asyncapi/send/lightTurnOn
in Line 25 does.
The example i made show what should work imo. If you clone it and run it you will see that the bundler will throw an error:
ResolverError: Error opening file "/home/example/asyncapi/bundler/schema.json"
ENOENT: no such file or directory, open '/home/example/asyncapi/bundler/schema.json'
All other scripts like gen-markdown
or validate
work, just bundle does not. I did note the details for each script in the readme of the branch.
I'll compare how different tools resolve paths.
Other tools never had to deal with the resolution of $ref
s or the creation of a stack of YAML/JSON files that needed to be merged together. They always operate on one solid, complete AsyncAPI Document in YAML or JSON format. That's why there's a difference between handling a pack of paths and keeping calm about being fed a list of paths from an iterated array.
Bundler
needs to develop its own approach to solve
https://asyncapi.slack.com/archives/CQVJXFNQL/p1721921263521989?thread_ts=1721883187.801609&cid=CQVJXFNQL
and I propose
{
base: 'index.yaml',
baseDir: 'docs/asyncapi',
traverseSubdirsFor: 'asyncapi.yaml',
}
With these options specified, the Bundler
will:
baseDir
for file traverseSubdirsFor
(using glob
, for example, in a way it's done here)traverseSubdirsFor
file a new current Node.js dir, resolve all $ref
s relative to traverseSubdirsFor
, repeat in case of a need (maybe creating a separate JS object with { 0: 'bundled.yaml from dir0', 1: 'bundled.yaml from dir1', 2: 'bundled.yaml from dir2', }
, maybe saving down a bundled.yaml
next to traverseSubdirsFor
file it just dereferenced, maybe both, maybe depending on a switch in the options
object)base
Parser
before outputbundled.yaml
if it passed pre-validationWithout traverseSubdirsFor
, the Bundler
will assume that it should work as usual, at a flat level, without traversing into subdirectories, only resolving $ref
s relative to the main YAML/JSON specified as a parameter.
@laupetin, @derberg, how does that look?
I am a bit confused about why you bring up a traverse subdirectories option 😅 For the issue here it is not really required as far as I saw it.
The reason why it is not required is that the bundler resolves the $refs before bundling. So you do not need to think about which folders apply to which $ref because all of them have been resolved at the bundling step already.
I did already propose a solution to the issue with the linked pull request that makes sure any loaded asyncapi document uses its own folder when resolving $refs. It is only afterwards that they are bundled. It fixes the issue in the example I have given. If i'm forgetting something and there is a reason why that approach doesnt work, please tell me. Right now im just confused about why that traversing logic would be necessary.
Without traverseSubdirsFor, the Bundler will assume that it should work as usual, at a flat level, without traversing into subdirectories, only resolving $refs relative to the main YAML/JSON specified as a parameter.
This is not what it does currently, it resolves $refs based on the current working directory by default, not based on the folder of the main yaml/json
@Laupetin
Your directory structure is potentially a new convention for structuring complex systems built on asynchronous APIs, where each operation is the responsibility of a separate team, up to having its own separate Git repository.
For now, there is no convention that a bundling tool should look exactly for a file asyncapi.yaml
in nested directories (thus traverseSubdirsFor
,) and there is no convention for how nested directories should even be named.
Should their naming repeat JSON Pointers' structure of the main AsyncAPI Document that describes the full system, so it could be parsed? Should there be a switch that lays down bundled YAMLs next to asyncapi.yaml
for gathering them later by a CI? How about DE-bundling when a tool cuts the main AsyncAPI Document to pieces, using JSON Pointers' structure to FORM nested directories?
To me, the matter raised in this issue is not a feat!:
because Bundler didn't reach v1.0.0
yet, but certainly not a small fix:
because it nudges to rethinking the very ideology of Bundler
.
Therefore, letting the current SIG group know about this idea just in case. @fmvilas @jonaslagoni @smoya
#
However, if I was fixing this issue as a plain bug, I would use one loop utilizing the tools I already have, without introducing new ones, and it would still be doing the same thing as https://github.com/asyncapi/bundler/pull/179
Please, copy&paste this code into your index.ts
from master
, compile, and check on your setup.
This functionality is implemented in Bundler
v0.6.0
.
This functionality is added in CLI
v2.3.1
.
Why do we need this improvement?
Resolving relative paths from asyncapi documents located in different folders is currently not possible in a way that would keep the scope of the single files locally.
For example i am trying to destructure our asyncapi spec into smaller documents to reduce complexity. To do this i tried to use the following folder structure:
index.yaml
would be the base file and our different operations would be split into separate files. Now when trying to bundle this into a single asyncapi spec i am running into a problem when trying to referenceschema.yaml
from anyasyncapi.yaml
via./schema.yaml
.By default it will take the current working directory which is try to resolve it to
docs/schema.yaml
. With the help of the--baseDir
arg I would only be able to set the baseDir todocs
. I would still have to referenceschema.yaml
via a path that is not relative to the files that define each operation (event-zero, event-one):receive/event-zero/schema.yaml
.While that works, it is easy to make a mistake when creating a new operation and it also makes it impossible to validate each document separately, since validating always assumes $refs relative to the document's directory.
How will this change help?
asyncapi generate fromTemplate
parsing expects the paths to be relative to the document without the possibility to specify a "baseDir"Screenshots
No response
How could it be implemented/designed?
Change directory to the folder of each asyncapi document when dereferencing any $refs, unless a
--baseDir
option is specified (to not break the functionality of it).Alternatively it is also possible to implement this in a non-breaking way in which it would only apply when a new option like
--path-relative-to-document
is specified in case a breaking change by changing default behaviour is undesired.🚧 Breaking changes
Yes
👀 Have you checked for similar open issues?
🏢 Have you read the Contributing Guidelines?
Are you willing to work on this issue?
Yes I am willing to submit a PR!