apache / incubator-devlake

Apache DevLake is an open-source dev data platform to ingest, analyze, and visualize the fragmented data from DevOps tools, extracting insights for engineering excellence, developer experience, and community growth.
https://devlake.apache.org/
Apache License 2.0
2.57k stars 516 forks source link

[Bug][Scope Config] Only the first 10 items are inheriting the scope config #7503

Closed slaterx closed 1 month ago

slaterx commented 4 months ago

Search before asking

What happened

Created a new project, added 100 data scopes from existing github connection in it:

image

However, only the first 10 are inheriting the scope config, all other 90 are showing N/A for scope config:

image

What do you expect to happen

All scopes have a valid scope config in it

How to reproduce

Anything else

We're troubleshooting why we are not seeing any numbers/data in DORA grafana charts, and we arrived at this situation.

Version

1.0.0-beta6

Are you willing to submit PR?

Code of Conduct

dosubot[bot] commented 4 months ago

Hey @slaterx! I'm Dosu, a friendly bot here to help you solve bugs, answer questions, and guide you in becoming a contributor. While you wait for a human maintainer, I'm here to assist! 😄

The issue you're encountering, where only the first 10 items are inheriting the scope config and the rest are showing "N/A," could be related to how the scope configurations are being processed and inherited in the codebase.

Potential Cause

The scope config is marked as "N/A" when the scopeConfigId is not provided or is undefined [1]. This might be happening due to a limitation or bug in the code that processes the scope configurations for multiple data scopes.

Relevant Code

The inheritance of scope configurations is implemented using Go's struct embedding feature. The GithubScopeConfig struct embeds the common.ScopeConfig struct, which allows it to inherit fields and methods from common.ScopeConfig [2].

Troubleshooting Steps

  1. Check the API Handling: Ensure that the API endpoints responsible for creating and updating scope configurations are correctly handling all the data scopes. The relevant functions are PostScopeConfig, PatchScopeConfig, GetScopeConfig, GetScopeConfigList, and DeleteScopeConfig [3].

  2. Review the Pipeline Plan Creation: The MakeDataSourcePipelinePlanV200 function in blueprint_v200.go handles multiple data scopes when creating a pipeline plan. Ensure that this function is correctly loading and applying the scope configurations for all data scopes [4].

  3. Logs and Error Messages: Check for any logs or error messages that might indicate issues with decoding or applying the scope configurations. Look for messages like "error in decoding scope config" or "no scope config with given id" [5].

Next Steps

By following these steps, you should be able to identify and resolve the issue causing only the first 10 items to inherit the scope config. If you need further assistance, feel free to ask!

#### Details **About Dosu** This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

klesh commented 4 months ago

Can you share screenshots of the connection detail page for those scopes that are NOT inheriting scope config?

slaterx commented 4 months ago

@klesh it's all in the same connection, not a new one. Installation has only one project with only one connection in it. And all repos in it have a configured scope in it.

github-actions[bot] commented 2 months ago

This issue has been automatically marked as stale because it has been inactive for 60 days. It will be closed in next 7 days if no further activity occurs.

github-actions[bot] commented 1 month ago

This issue has been closed because it has been inactive for a long time. You can reopen it if you encounter the similar problem in the future.