[BUG] Intermittent 403 RequestError in build-dashboard.js script

anshgoyalevil commented 6 months ago

Describe the bug.

We have a build-dashboard.js script in place that runs every midnight via the GitHub Actions Cron Job defined at https://github.com/asyncapi/website/actions/runs/8730442751/workflow#L7 along with 2 other scripts build-meetings.js and build-newsroom-videos.js.

The AsyncAPI bot then creates the PR with the resulting file changes.

There's some error in the script due to which we are reaching GitHub API limits too soon. The reported error is: https://github.com/asyncapi/website/actions/runs/8501984640/job/23285563590#step:6:54

RequestError [HttpError]: You have exceeded a secondary rate limit. Please wait a few minutes before you try again. If you reach out to GitHub Support for help, please include the request ID 7C54:8291:1619423:2304A79:6609FFC4.

Expected behavior

There must not be any such error. Either try to find an alternate method for replacing the failing code in build-dashboard.js or fix the existing approach.

Screenshots

How to Reproduce

Check out the workflow runs here: https://github.com/asyncapi/website/actions/workflows/regenerate-meetings-and-videos.yml

🥦 Browser

None

👀 Have you checked for similar open issues?

[X] I checked and didn't find similar issue

🏢 Have you read the Contributing Guidelines?

[X] I have read the Contributing Guidelines

Are you willing to work on this issue ?

None

NightFury742 commented 4 months ago

hey @anshgoyalevil can I work on this?

anshgoyalevil commented 4 months ago

@NightFury742 I am not sure if this bug still persists. Feel free to check if it exists and resolve :)

Wck-iipi commented 1 month ago

@anshgoyalevil I was looking at this issue and found out that this workflow is not running for quite some time now. It is giving the following error:

> @asyncapi/website@0.1.0 generate:dashboard
> node scripts/dashboard/build-dashboard.js

/home/runner/work/website/website/scripts/dashboard/build-dashboard.js:4
const *** Promise *** = require('node-fetch');
                    ^

Error [ERR_REQUIRE_ESM]: require() of ES Module /home/runner/work/website/website/node_modules/node-fetch/src/index.js from /home/runner/work/website/website/scripts/dashboard/build-dashboard.js not supported.
Instead change the require of index.js in /home/runner/work/website/website/scripts/dashboard/build-dashboard.js to a dynamic import() which is available in all CommonJS modules.
    at Object.<anonymous> (/home/runner/work/website/website/scripts/dashboard/build-dashboard.js:4:21) ***
  code: 'ERR_REQUIRE_ESM'

(It is giving error for 4 months: ) I would like to work on this issue, however, should I create new issue for this error as I couldn't find a issue for above error?

aeworxet commented 1 month ago

Bounty Issue's service comment

Text labels: bounty/2024-Q4, bounty/advanced, bounty/coding, bounty/upgraded First assignment to regular contributors: 2024-09-20 00:00:00 UTC+12:00 End Of Life after: 2024-10-31 23:59:59 UTC-12:00

@asyncapi/bounty_team

The Bounty Program is not a Mentorship Program. The accepted level of Bounty Program Participants is Middle/Senior.

Regular contributors should explain in meaningful words how they are going to approach the resolution process when expressing a desire to work on this Bounty Issue.

aialok commented 1 month ago

Hi team,

I would like to take on this issue. I have good experience with GitHub Actions from my GSoC work, where I built the CI/CD pipeline for the JSON Schema website.

Thank you : )

aialok commented 1 month ago

My Plan :

Quick Fix: We can downgrade the node-fetch package it will work.

But I will try to find better solution and alternative to work with latest version.

Also we need to update github action workflows since we are using older version of node

I have tried fixing with degrading version it is working fine but we need to test it in github actions environment and find a good solutions. Also current github action packages are not up to date that need to fix.

Thank you : )

Wck-iipi commented 1 month ago

@aeworxet I would like to get assigned for this issue as I had already started working on it 4 days before and would be completed soon. Thanks.

aeworxet commented 1 month ago

@Wck-iipi

Please refrain from starting work until being assigned, trying to impose the fact that you must be assigned simply because you already did half the work you weren't assigned to in the first place.

On the other hand, you can describe in plain words what you have already done and what you are planning to do next without sharing the code. Probably this will increase your chances of being assigned to this Bounty Issue.

In any case, before sharing your approach, please specify your full name in your GitHub profile.

sambhavgupta0705 commented 1 month ago

Ref: https://github.com/asyncapi/website/issues/3036

jerensl commented 1 month ago

@aeworxet I think I'm already made a fix for the node-fetch problem in this PR https://github.com/asyncapi/website/pull/3038/files#diff-cfbf694e8e8ffe40a57d3b7d5357d38fde7c3a21b22f9d6c74e4cf583f4d3a84, but the issues is seems comeback.

For the rate limiting, I think we can reduce the frequence of the cron jobs, reduce retry etc.

I also like to work on this issues if this still exist under bouty program

Some of my past contribution: https://github.com/asyncapi/modelina/pull/2067 https://github.com/asyncapi/modelina/pull/2028 https://github.com/asyncapi/modelina/pull/2031

jerensl commented 1 month ago

My Plan :

Quick Fix: We can downgrade the node-fetch package it will work.

But I will try to find better solution and alternative to work with latest version.

Also we need to update github action workflows since we are using older version of node

I have tried fixing with degrading version it is working fine but we need to test it in github actions environment and find a good solutions. Also current github action packages are not up to date that need to fix.

Thank you : )

For the issues you are mention, Im already made PR for that and this PR should fix the issues regarding of node-fetch https://github.com/asyncapi/website/pull/3099/files

aialok commented 1 month ago

I dont think reducing number of cron job will resolve github rate limiting. The problem is not because of github rate limiting but because of node fetch package.

We are still using v3 which have some esm import error.

https://github.com/asyncapi/website/blob/0c6722614b437312090ab98dcd697f1c814adaec/scripts/dashboard/build-dashboard.js#L4

Here is fix:

https://github.com/asyncapi/website/issues/2894#issuecomment-2353706914

I would love to work of this issue : )

aialok commented 1 month ago

My Plan : Quick Fix: We can downgrade the node-fetch package it will work. But I will try to find better solution and alternative to work with latest version. Also we need to update github action workflows since we are using older version of node I have tried fixing with degrading version it is working fine but we need to test it in github actions environment and find a good solutions. Also current github action packages are not up to date that need to fix. Thank you : )

For the issues you are mention, Im already made PR for that and this PR should fix the issues regarding of node-fetch https://github.com/asyncapi/website/pull/3099/files

Yes, This PR might fix the issue : )

aeworxet commented 1 month ago

I think the issue has changed a bit since the time it was introduced.

https://github.com/asyncapi/website/actions/runs/10913294253/job/30289538132

/home/runner/work/website/website/scripts/dashboard/build-dashboard.js:4
const *** Promise *** = require('node-fetch');
                    ^

Error [ERR_REQUIRE_ESM]: require() of ES Module /home/runner/work/website/website/node_modules/node-fetch/src/index.js from /home/runner/work/website/website/scripts/dashboard/build-dashboard.js not supported.
Instead change the require of index.js in /home/runner/work/website/website/scripts/dashboard/build-dashboard.js to a dynamic import() which is available in all CommonJS modules.
    at Object.<anonymous> (/home/runner/work/website/website/scripts/dashboard/build-dashboard.js:4:21) ***
  code: 'ERR_REQUIRE_ESM'
***

and will require a little more than simple juggling with cron jobs and retries.

aialok commented 1 month ago

I think the issue has changed a bit since the time it was introduced.

https://github.com/asyncapi/website/actions/runs/10913294253/job/30289538132

/home/runner/work/website/website/scripts/dashboard/build-dashboard.js:4
const *** Promise *** = require('node-fetch');
                    ^

Error [ERR_REQUIRE_ESM]: require() of ES Module /home/runner/work/website/website/node_modules/node-fetch/src/index.js from /home/runner/work/website/website/scripts/dashboard/build-dashboard.js not supported.
Instead change the require of index.js in /home/runner/work/website/website/scripts/dashboard/build-dashboard.js to a dynamic import() which is available in all CommonJS modules.
    at Object.<anonymous> (/home/runner/work/website/website/scripts/dashboard/build-dashboard.js:4:21) ***
  code: 'ERR_REQUIRE_ESM'
***

and will require a little more than simple juggling with cron jobs and retries.

Yes, exactly. The problem is with node-fetch package.

jerensl commented 1 month ago

I think the issue has changed a bit since the time it was introduced.

https://github.com/asyncapi/website/actions/runs/10913294253/job/30289538132

/home/runner/work/website/website/scripts/dashboard/build-dashboard.js:4
const *** Promise *** = require('node-fetch');
                    ^

Error [ERR_REQUIRE_ESM]: require() of ES Module /home/runner/work/website/website/node_modules/node-fetch/src/index.js from /home/runner/work/website/website/scripts/dashboard/build-dashboard.js not supported.
Instead change the require of index.js in /home/runner/work/website/website/scripts/dashboard/build-dashboard.js to a dynamic import() which is available in all CommonJS modules.
    at Object.<anonymous> (/home/runner/work/website/website/scripts/dashboard/build-dashboard.js:4:21) ***
  code: 'ERR_REQUIRE_ESM'
***

and will require a little more than simple juggling with cron jobs and retries.

To give some context I made PR to fix this issues which happened first on the root of build-newsroom-videos.js but wasn't aware there other node fetch inside build-dashboard.js and made follow up PR but doesn't got merge since several month

aialok commented 1 month ago

I think the issue has changed a bit since the time it was introduced. https://github.com/asyncapi/website/actions/runs/10913294253/job/30289538132

/home/runner/work/website/website/scripts/dashboard/build-dashboard.js:4
const *** Promise *** = require('node-fetch');
                    ^

Error [ERR_REQUIRE_ESM]: require() of ES Module /home/runner/work/website/website/node_modules/node-fetch/src/index.js from /home/runner/work/website/website/scripts/dashboard/build-dashboard.js not supported.
Instead change the require of index.js in /home/runner/work/website/website/scripts/dashboard/build-dashboard.js to a dynamic import() which is available in all CommonJS modules.
    at Object.<anonymous> (/home/runner/work/website/website/scripts/dashboard/build-dashboard.js:4:21) ***
  code: 'ERR_REQUIRE_ESM'
***

and will require a little more than simple juggling with cron jobs and retries.

Yes, exactly. The problem is with node-fetch package.

Here is fix. https://github.com/asyncapi/website/issues/2894#issuecomment-2353706914

I tested it in my local it is working fine but need to check in github actions environment.

akshatnema commented 1 month ago

Hey @aeworxet , I want to work on this issue as part of the Bounty program Q4. Do assign it to me.

aeworxet commented 1 month ago

@akshatnema is an AsyncAPI Maintainer specified in https://raw.githubusercontent.com/asyncapi/community/master/MAINTAINERS.yaml, so they fall under the first category in the prioritization list.

aeworxet commented 1 month ago

Bounty Issue's Timeline

Complexity Level	Assignment Date (by GitHub)	Start Date (by BP Rules)	End Date (by BP Rules)	Draft PR Submission	Final PR Merge Start	Final PR Merge End
Medium	2024-09-19	2024-10-07	2024-11-17	2024-10-20	2024-11-03	2024-11-17

Please note that the dates given represent deadlines, not specific dates, so if the goal is reached sooner, it's better.

Keep in mind the responsibility for violations of the Timeline.

akshatnema commented 1 month ago

@aeworxet There are two more GitHub issues that are subtly related to this one and still need to be done:

https://github.com/asyncapi/website/issues/3132 https://github.com/asyncapi/website/issues/3008

I would like to append them to this Bounty Issue and reclassify it to Advanced.

aeworxet commented 1 month ago

Upon request of the AsyncAPI Maintainer, who is also the Bounty Program Participant (@akshatnema), the Complexity Level of this Bounty Issue was reclassified to Advanced (upgraded) and its Timeline changed.

Bounty Issue was Reclassified and its Timeline changed

Complexity Level	Assignment Date (by GitHub)	Start Date (by BP Rules)	End Date (by BP Rules)	Draft PR Submission	Final PR Merge Start	Final PR Merge End
Advanced	2024-09-19	2024-10-07	2024-12-01	2024-10-27	2024-11-17	2024-12-01

Please note that the dates given represent deadlines, not specific dates, so if the goal is reached sooner, it's better.

Keep in mind the responsibility for violations of the Timeline.

aeworxet commented 4 weeks ago

Changed this Bounty Issue into an Aggregated Bounty Issue https://github.com/asyncapi/website/issues/3247 that accumulates

in it, for the sake of accounting simplification.

asyncapi / website