search

asyncapi / parser-js

AsyncAPI parser for Javascript (browser-compatible too).
Apache License 2.0
116 stars 91 forks source link

Integrate smoya/multi-parser-js into asyncapi/parser-js #963

Open smoya opened 5 months ago

smoya commented 5 months ago

Context

@smoya/multi-parser is a library that allows users to parse an AsyncAPI document and get an output model based on a desired Parser-API version. Additionally, it allows to convert models from one Parser-API version to another.

This tool is required by some AsyncAPI tools that use the Parser-JS and have to send the parsed document model to another library, which could be compatible with a different Parser-API version than the one used in that particular version of the Parser-JS. You have several examples of usage here:

Current issues

1. 👤 Ownership

@smoya/multi-parser is a library hosted in a GH repository under my particular ownership (@smoya). Not a big issue per se, but I don't wanna be the only maintainer of such a tool. I thought on donating this to AsyncAPI org (TSC would need to vote first) but the reality is that won't be needed, see next point:

2. 🔃 Release is not in sync with Parser-JS

Every new Parser-JS release is not triggering an update on the @smoya/multi-parser. I agree I could configure a bot like Renovate (Dependabot doesn't allow NPM package aliases, and this package makes use of them) to try update those. But still we have two issue with that situation:

  1. Whenever a new major Parser-API version gets released, the need of small additions into the @smoya/multi-parser code will be needed. See https://github.com/smoya/multi-parser-js/blob/main/src/parse.ts#L45-L53. Could we automate it somehow by changing the code? Perhaps yes, but not ideal and probably not trustable in the long term. At this moment, this includes manual work and be always careful when a new version goes out.
  2. For a certain period of time after a new release of Parser-JS, @smoya/multi-parser will be outdated and users of the library who also use Parser-JS as dependency could have runtime errors due to not using the same version of the models (different Parser-JS versions!). Of course, solved when Renovate updates in case it's not a major Parser-API version change.

The solution I propose

To integrate @smoya/multi-parser code into the Parser-JS repository, and to release two packages on each Parser-JS release:

In that way, everything gets in sync, and also, by adding tests, we could ensure new versions of the Parser-JS are working with the @asyncapi/multi-parser package. The same when we adopt a new version of the Parser-API in the Parser-JS, support in @asyncapi/multi-parser will be added in the same PR.

That means, we should have a monorepo model in this repository. Due to the fact @asyncapi/multi-parser has as dependency the Parser-JS but not only once, but once for each Parser-API version (atm 3 aliased Parser-JS deps pointing to different version of the Parser-JS), plus the fact the library will be used only by few particular libraries and most users won't need it ever, @asyncapi/multi-parser should be released completely as a separate package.

Examples of monorepo within AsyncAPI can be found here:

Some discussions in Slack about that topic:

BTW, since @smoya/multi-parser is under Apache 2.0 license, copying code right directly into the Parser-JS is completely legal, and since Parser-JS uses the same license, no need to specify any atribution of change on the license. So no need to even first do the donation to AsyncAPI GH org :) Additionally, I'm giving my written approval to do whatever is needed with that code so it can be integrated into the Parser-JS. Once that's done, and all dependants update to @asyncapi/multi-parser, I will completely archive @smoya/multi-parser in favor of @asyncapi/multi-parser.

cc @jonaslagoni @magicmatatjahu

jonaslagoni commented 5 months ago

Makes sense to me 👍

aeworxet commented 5 months ago

Bounty Issue's service comment

Text labels: bounty/2024-Q2, bounty/advanced, bounty/coding First assignment to third-party contributors: 2024-03-22 00:00:00 UTC+12:00 End Of Life: 2024-08-31 23:59:59 UTC-12:00

@asyncapi/bounty_team

Vishal2002 commented 5 months ago

@smoya I think you should ask the Asyncapi org to add this issue under GSOC 2024 idealist. It would be a good contribution to have in the parser-js.

smoya commented 5 months ago

@smoya I think you should ask the Asyncapi org to add this issue under GSOC 2024 idealist. It would be a good contribution to have in the parser-js.

This is part now of the AsyncAPI Bounty Program 2024 Q2. Additionally, the GSOC 2024 ideas submission period finished last month, and we got accepted.

derberg commented 5 months ago

@smoya if that works, I guess we can start thinking about pulling other projects inside parser-js for better maintainance and cooperation - like avro, protobuf, openapi and raml data parsers?

and then we can also work on something that @jonaslagoni brought up in past, that DX of using parser with schema parsers is hard, so we could enable publishing of parser-js on steroids, with schema parsers included by default (while still publishing parser alone of course, lightweight for web)

smoya commented 5 months ago

@smoya if that works, I guess we can start thinking about pulling other projects inside parser-js for better maintainance and cooperation - like avro, protobuf, openapi and raml data parsers?

It makes sense to me, yeah.

and then we can also work on something that @jonaslagoni brought up in past, that DX of using parser with schema parsers is hard, so we could enable publishing of parser-js on steroids, with schema parsers included by default (while still publishing parser alone of course, lightweight for web)

+1. I wasn't aware of this suggestion but i really like it. I've found myself involved in the issue of having to manually load the schema parsers all the time so I believe it will be so helpful for lot of users who just want everything to work at once.

derberg commented 5 months ago

+1. I wasn't aware of this suggestion but i really like it. I've found myself involved in the issue of having to manually load the schema parsers all the time so I believe it will be so helpful for lot of users who just want everything to work at once.

yeah, I see the same with semantic-versioning that at some point of time, decided to include in main package a number of default, more used plugins - so managing dependencies is now much much better

Vishal2002 commented 5 months ago

I want to give a try to this issue and want to work on it.

smoya commented 5 months ago

Hi @Vishal2002, thanks for showing interest. Let's wait at least a day since more contributors could jump in as well. As per the Bounty rules:

Assignment of the Bounty Issue on GitHub to users that fall under the second and third categories is performed not earlier than three calendar days after the addition of the GitHub label bounty according to GitHub's timestamp. Source: https://github.com/asyncapi/community/pull/897/files#diff-25ecb20a61754c225d6511ca08d7e7c9a14b9ca5a93e89bd42331e51c9ebf26dR39

In your case, you fall under the third category.

Toyin5 commented 5 months ago

@smoya I want to work on this issue

smoya commented 5 months ago

@smoya I want to work on this issue Cool! 🚀

Same answer as the one I provided to @Vishal2002 in https://github.com/asyncapi/parser-js/issues/963#issuecomment-2009341834

derberg commented 5 months ago

@smoya in case you don't get experienced candidates for this one, maybe you can wait for @ayushnau (maybe he will be interested) as he is going to work now on basically the same (with small differences) https://github.com/asyncapi/generator/issues/1044 - so monorepo in generator. Once he finish it, he can then do the same here maybe. Just an idea, I did not even check with @aeworxet if it is possible but I guess yes 🤔 you do not have to decide today or tomorrow, you can decide in a month

smoya commented 5 months ago

@smoya in case you don't get experienced candidates for this one, maybe you can wait for @ayushnau (maybe he will be interested) as he is going to work now on basically the same (with small differences) asyncapi/generator#1044 - so monorepo in generator. Once he finish it, he can then do the same here maybe. Just an idea, I did not even check with @aeworxet if it is possible but I guess yes 🤔 you do not have to decide today or tomorrow, you can decide in a month

Yeah, good advise. I'm still evaluating each candidate individually in the meantime.

smoya commented 5 months ago

Whats your opinion on this @ayushnau ? If you are willing to work on this, do you believe you could become available at any time soon? When approx?

Thanks! 🙏

ayushnau commented 5 months ago

@smoya I'm excited about the opportunity to collaborate on this with you, I would love to work on this. Looking forward to it!

ayushnau commented 5 months ago

I will be available starting from next Monday. to pay full attention to this.

smoya commented 5 months ago

@ayushnau are you gonna work on this then? If so, when are you finally planning to do it? Thanks!

ayushnau commented 5 months ago

i am on it once it got assigned to me.

smoya commented 5 months ago

i am on it once it got assigned to me.

As spoken in Slack, you are working in https://github.com/asyncapi/generator/issues/1044 in parallel. You mentioned you need a couple of weeks to finish such a task. Let's wait 2 weeks until I assign this issue to you. Please @ayushnau ping me max on the 18 of April so I can assign this task to you.

ayushnau commented 5 months ago

sure will do that.

ayushnau commented 4 months ago

@smoya I would love to work on this please assign this issue to me.

smoya commented 4 months ago

@asyncapi/bounty_team issue got assigned today to @ayushnau

aeworxet commented 4 months ago

Bounty Issue's Timeline

Complexity Level Assignment date (by GitHub) Start date (by BP rules) End date (by BP rules) Draft PR submission Final PR submission Final PR merge
Advanced 2024-04-18 2024-04-22 2024-06-14 2024-05-10 2024-05-31 2024-06-14
Please note that the dates given represent deadlines, not specific dates, so if the goal is reached sooner, it's better.
Keep in mind the responsibility for violations of the Timeline.
smoya commented 3 months ago

Due to the fact Ci has been broken since long ago (we were not aware), and considering it took me some time (days) figuring out the reason, I believe @ayushnau should request an extension of 1 week for this Bounty Program delivery.

cc @aeworxet

aeworxet commented 3 months ago

Upon request of the AsyncAPI Maintainer, who is responsible for the resolution of the Bounty Issue from the AsyncAPI's side (@smoya), all remaining target dates of the Bounty Issue's Timeline are extended by one calendar week.

Bounty Issue's Timeline Extended

Complexity Level Assignment date (by GitHub) Start date (by BP rules) End date (by BP rules) Draft PR submission Final PR submission Final PR merge
Advanced 2024-04-18 2024-04-22 2024-06-21 2024-05-10 2024-05-31 2024-06-21
Please note that the dates given represent deadlines, not specific dates, so if the goal is reached sooner, it's better.
Keep in mind the responsibility for violations of the Timeline.
ayushnau commented 2 months ago

@aeworxet @smoya

I am writing to inform you of a recent change in our release process for the GitHub action. The release process, which was previously managed using semantic release, has now been transitioned to changeset. This unexpected shift has caused some delays, as I was not fully prepared for the new process. Consequently, I kindly request an extension of two weeks to accommodate this transition and ensure a smooth and accurate release.

Thank you for your understanding and consideration.

smoya commented 2 months ago

@aeworxet @smoya

I am writing to inform you of a recent change in our release process for the GitHub action. The release process, which was previously managed using semantic release, has now been transitioned to changeset. This unexpected shift has caused some delays, as I was not fully prepared for the new process. Consequently, I kindly request an extension of two weeks to accommodate this transition and ensure a smooth and accurate release.

Thank you for your understanding and consideration.

I agree the uncertainty nature of this issue, the knowns VS unknowns, and the several areas this issue affects (code, CI, CD) makes a difficult and, somehow, unpredictable one. Even though we had a model to follow (Studio case), we (including me as maintainer) did assumptions wrongly; later found we were missing some changes such as the replacement of semantic-release by changesets (a change that it is not so obvious) and how that change affects our release pipeline.

Being honest, I had some doubts few days ago if this task should be given an extension or not, but atm I truly believe it should get it.

@aeworxet has the last words.

aeworxet commented 2 months ago

Upon request of the Bounty Program Participant (@ayushnau), all remaining target dates of the Bounty Issue's Timeline are extended by two calendar weeks.

Bounty Issue's Timeline Extended

Complexity Level Assignment date (by GitHub) Start date (by BP rules) End date (by BP rules) Draft PR submission Final PR submission Final PR merge
Advanced 2024-04-18 2024-04-22 2024-07-05 2024-05-10 2024-06-21 2024-07-05
Please note that the dates given represent deadlines, not specific dates, so if the goal is reached sooner, it's better.
Keep in mind the responsibility for violations of the Timeline.
smoya commented 1 month ago

@ayushnau After merging https://github.com/asyncapi/parser-js/pull/1036 and related PRs, the release of both packages happened. I tested the @asyncapi/parser (basic stuff) and seems to work, however, when running a simple test for the @asyncapi/multi-parser, the following errors appears:

(node:55793) Warning: To load an ES module, set "type": "module" in the package.json or use the .mjs extension.
(Use `node --trace-warnings ...` to show where the warning was created)
/Users/smoya/dev/asyncapi-parser-example/node_modules/@asyncapi/parser/esm/index.js:1
import { DiagnosticSeverity } from '@stoplight/types';
^^^^^^

SyntaxError: Cannot use import statement outside a module
    at wrapSafe (internal/modules/cjs/loader.js:1001:16)
    at Module._compile (internal/modules/cjs/loader.js:1049:27)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1114:10)
    at Module.load (internal/modules/cjs/loader.js:950:32)
    at Function.Module._load (internal/modules/cjs/loader.js:790:12)
    at Module.require (internal/modules/cjs/loader.js:974:19)
    at require (internal/modules/cjs/helpers.js:93:18)
    at Object.<anonymous> (/Users/smoya/dev/asyncapi-parser-example/node_modules/@asyncapi/multi-parser/cjs/parse.js:6:17)
    at Module._compile (internal/modules/cjs/loader.js:1085:14)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1114:10)
ayushnau commented 1 month ago

@smoya working on it. will update you once i fix it.

smoya commented 1 month ago

This issue is now technically solved. Thanks @ayushnau for your work!

Cc @aeworxet

aeworxet commented 1 month ago

After a set of delays that were beyond the control of the Bounty Program Participant https://github.com/asyncapi/parser-js/pull/1036#issuecomment-2210481751 https://github.com/asyncapi/parser-js/pull/1036#issuecomment-2214704347 https://github.com/asyncapi/parser-js/pull/1036#issuecomment-2214973049 https://github.com/asyncapi/parser-js/pull/1036#issuecomment-2228059461 https://github.com/asyncapi/parser-js/pull/1036#issuecomment-2238512092 https://github.com/asyncapi/parser-js/pull/1036#pullrequestreview-2193833513 https://github.com/asyncapi/parser-js/pull/1036#issuecomment-2245112659 https://github.com/asyncapi/parser-js/issues/963#issuecomment-2245364804

Bounty Issue Completed 🎉

@ayushnau, please go to the AsyncAPI page on Open Collective and submit an invoice for USD 400.00 with the expense title Bounty parser-js#963, tag bounty, and full URL of this Bounty Issue in the description.