dosubot[bot]
commented
1 month ago Hey there, @Nickcw6! I'm here to help you with any bugs, questions, or contributions you might have. Let's tackle this issue together!
The issue you're experiencing with the CircleCI plugin collecting data only from the past 24 hours, regardless of the Time Range setting, is likely due to the timeAfter parameter not being correctly set. Here are the steps to ensure the timeAfter parameter is set correctly and to collect data from the full specified time range:
- Create a Project: Ensure you have created a project as described in the documentation.
- Add a CircleCI Connection: Add the CircleCI connection to the project.
- Set the Sync Policy:
- Go to the Sync Policy settings.
- Set the
Data Time Rangeto the desired time range instead of the default past 24 hours.
Example:
### Step 2.3 - Set the Sync Policy
There are three settings for Sync Policy:
- **Data Time Range**: You can select the time range of the data you wish to collect. The default is set to the past six months. Ensure this is set to the full time range you need.
- **Sync Frequency**: Choose how often you would like to sync your data by selecting a sync frequency option or enter a cron code to specify your preferred schedule.
- **Skip Failed Tasks**: Choose to skip failed tasks to avoid spending more time running the pipeline all over again.- Run the Plugin with Correct Parameters: Ensure that the
timeAfterparameter is set correctly when running the plugin. For example:
cmd := &cobra.Command{Use: "CircleCi"}
connectionId := cmd.Flags().Uint64P("connection", "c", 0, "CircleCi connection id")
projectSlug := cmd.Flags().StringP("project", "p", "", "CircleCi project slug")
timeAfter := cmd.Flags().StringP("timeAfter", "a", "", "collect data that are created after specified time, ie 2006-01-02T15:04:05Z")
cmd.Run = func(cmd *cobra.Command, args []string) {
runner.DirectRun(cmd, args, PluginEntry, map[string]interface{}{
"connectionId": *connectionId,
"projectSlug": *projectSlug,
}, *timeAfter)
}
runner.RunCmd(cmd)By setting the timeAfter parameter correctly and starting the data collection, the CircleCI plugin should collect data from the full specified time range [1][2][3][4].
Additionally, ensure that there are no API rate limits or other issues affecting the data collection. The CircleCI plugin handles pagination and rate limits, but if you encounter specific error messages or logs, they can provide insights into why data collection might be limited [5][6][7][8].
To continue talking to Dosu, mention @dosu.
klesh
Nickcw6
Search before asking
What happened
When running a data collection for a CircleCI connection, data only appears to be collected from the past <24 hours, irrespective of what
Time Rangeis set to. Same behaviour observed in 'full refresh mode' & normal data collection.Seemed to have slightly differing behaviour each time I tried - when originally raised on Slack only the last ~3 hours of data was collected, however when reproducing again to raise this issue, seems to now have data from the past ~24 hours.
E.g. time frequency set to start of the year, then checking the
_tool_circleci_workflowtable:Only 18 workflows are identified, the earliest of which occurring at
2024-07-15 10:29:09.000. I would expect to see many more rows dating back to2024-01-01.CircleCI pipeline task logs:
Also have Github and Jira data connections running within the same pipeline, and data is pulled through as expected for both of these plugins.
EDIT: What is actually happening is only 20 pipelines are being collected from the CircleCI API response (ie. the first page). This then has a knock-on effect throughout the workflows and jobs tables.
What do you expect to happen
Data is collected from the full specified time range, e.g. starting from
2024-01-01(or whenever specified).How to reproduce
_tools_circleci_workflows,_tools_circleci_pipelinesor_tools_circleci_jobstables for expected row count, and earlieststarted_atorcreated_attimestamp (see below)Anything else
As an aside (but potentially related) - I notice there are discrepancies between the column names across the three CircleCI tool tables, e.g.
_tools_circleci_workflows-created_atis the timestamp the workflow was triggered in CircleCI. There is no other column which could represent the start of the workflow in CircleCI._tools_circleci_jobs-created_atis the timestamp the row was created in the DevLake DB, andstarted_atis the CircleCI timestamp._tools_circleci_pipelines-created_atis again the timestamp of DevLake DB creation. There iscreated_date, but this always seems to beNULL. As with the workflows table, there doesn't appear to be any column which represents the starting timestamp in CircleCI.Version
v1.0.0
Are you willing to submit PR?
Code of Conduct