[Question][REST API] I can't find a way to attach Connection to DevLake Project.

[kamiyariko]

I successfully created a DevLake project and my GitLab connection using the following APIs: POST: /projects POST: /plugins/gitlab/connections

Now, I'm trying to attach the connection to the project, but I couldn't figure out which API to use from the manual and Swagger documentation. Could someone tell me which API I should use?

[dosubot[bot]]

Hey @kamiyariko! 👋 I'm here to help you with any bugs or questions you have. I'm a bot that can assist with solving issues and providing information.

You should use the "TestConnection" API to attach the GitLab connection to the DevLake project. Here are the steps:

1. Test the GitLab Connection: Use the following API to test the connection:

   POST /plugins/gitlab/test

2. Add the GitLab Connection to the Project: After creating a project, you can add a previously configured GitLab connection to the project and select the boards for which you wish to collect the data.

For more detailed steps:

1. Add Data Connections: On the Connections page, select GitLab and create a new connection.
2. Authentication: Provide a unique connection name, select GitLab Cloud or Server, enter the endpoint URL, and provide your GitLab personal access token.
3. Test and Save Connection: Click Test Connection, and if successful, click Save Connection to add the connection.
4. Add Data Scopes: Select the GitLab repositories you want to collect data from.
5. Create a Project: Visit the Project page and create a new project.
6. Add a GitLab Connection: Add the previously configured GitLab connection to the project and select the boards for data collection.
7. Set the Sync Policy: Configure the data time range, sync frequency, and skip failed tasks options.
8. Start Data Collection: Click on "Collect Data" to start collecting data for the project ^{[1][2][3][4][5]}.

To continue talking to Dosu, mention @dosu.

[kamiyariko]

curl -X 'POST' \ 'https://XXXXX/api/plugins/gitlab/test' \ -H 'accept: application/json' \ -H 'Content-Type: application/json' \ -d '{ "endpoint": "https://XXXXX/api/v4/", "proxy": "string", "rateLimitPerHour": 0, "token": "XXXXX" }'

{"code":0,"success":true,"message":"success","causes":null,"data":null,"Connection":{"endpoint":"https://XXXXX/api/v4/","proxy":"string","rateLimitPerHour":0,"token":"XXXXXXXXXX"}}

The command you executed returned the following result, but no connection was attached to the project.

We want to configure settings using the API instead of the GUI. Have such features been implemented or will be implemented?

[dosubot[bot]]

Yes, features for attaching a GitLab connection to a DevLake project using the API have been implemented. You can manage GitLab connections via several API endpoints. To attach an existing GitLab connection to a DevLake project, you should use the following endpoint:

POST /projects/{projectId}/connections/{connectionId}

This endpoint allows you to add a previously configured GitLab connection to a specific DevLake project. Here is an example of how to use it:

curl -X 'POST' \
  'https://your-devlake-instance/api/projects/{projectId}/connections/{connectionId}' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "name": "Your Connection Name",
  "endpoint": "https://gitlab.com/api/v4/",
  "token": "Your Personal Access Token",
  "proxy": "Your Proxy URL (optional)",
  "rateLimitPerHour": 12000
}'

Make sure to replace {projectId} and {connectionId} with the actual IDs of your project and connection, respectively ^[1][2].

To continue talking to Dosu, mention @dosu.

[d4x1]

@kamiyariko

To attach a connection to an existing project, I think these steps are neccessary:

1. create a blueprint via {api}/blueprints in your project, get a blueprint_id.
2. post your connection to projects via {api}/blueprints/{blueprint_id}, because blueprint_id is belongs to your project, so connections can be added successfully.

You can view the payload using browser's dev tools.

[kamiyariko]

@dosu I tried it, but it returned 404. Sorry, I forgot to mention the version. We use v1.0.0-beta10.

[dosubot[bot]]

To resolve the 404 error when trying to attach a GitLab connection to a DevLake project using the API endpoint POST /projects/{projectId}/connections/{connectionId} in DevLake version v1.0.0-beta10, ensure the following steps are correctly followed:

1. Create a Project: Ensure you have created a project in DevLake. You can do this by visiting the Project page from the side menu and following the instructions on the user interface.

2. Add a GitLab Connection: Make sure you have added a GitLab connection correctly. This involves:

   • Providing a unique connection name.
   • Selecting the correct GitLab version (Cloud or Server).
   • Entering a valid REST API endpoint URL.
   • Using a valid GitLab personal access token with the necessary permissions (api or read_api).

3. Verify Endpoint URL: Ensure the endpoint URL is correct and ends with a /. For GitLab Cloud, it should be https://gitlab.com/api/v4/. For GitLab Server, it should be in the format https://gitlab.example.com/api/v4/.

4. Check Permissions: Verify that your GitLab user has the appropriate permissions and is not a Guest role in the GitLab projects you wish to collect data from.

5. Add Data Scopes: Ensure you have selected the GitLab repositories you want to collect data from and that they are not excluded due to guest permissions or being archived.

6. Test and Save Connection: After configuring the connection, click Test Connection to ensure it is successful, then click Save Connection.

If all the above steps are correctly followed and you still encounter a 404 error, it might be due to an incorrect projectId or connectionId in the API request. Double-check that the IDs used in the endpoint are correct and exist in your DevLake setup.

Example API request:

curl -X POST http://localhost:4000/api/projects/{projectId}/connections/{connectionId} \
-H "Content-Type: application/json" \
-d '{
  "data": {
    "projectId": "{projectId}",
    "connectionId": "{connectionId}"
  }
}'

Replace {projectId} and {connectionId} with the actual IDs from your DevLake setup.

Additionally, ensure that the connection has been tested successfully using the TestConnection endpoint provided by DevLake. This can help verify that the connection details are correct and that the necessary permissions are in place. If the connection test fails, it will provide specific error messages that can help diagnose the issue ^[1][2].

To continue talking to Dosu, mention @dosu.

[d4x1]

@kamiyariko dosu is a bot, you can try my advice.

[jensrotne]

I just had the same issue when I tried to POST a new project, and it seems like a bug in the projects API endpoint to me.

I am using the v1.0.0-rc1 image.

As @d4x1 said, then you first need to create the project using the POST /api/projects endpoint, and then patch the blueprint afterwards using the PATCH /api/blueprints/{blueprintId} endpoint. The blueprintId is the blueprint identifier you get back from the earlier POST request to the projects endpoint.

[kamiyariko]

Thank you for your excellent advice. @d4x1 @jensrotne I created project by this command,

curl -X 'POST' \
  'https://xxxxx/api/projects' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "blueprint": {
    "afterPlan": [
      [
        {}
      ]
    ],
    "beforePlan": [
      [
        {}
      ]
    ],
    "connections": [
      {
        "connectionId": 5,
        "pluginName": "string",
        "scopes": [
          {
            "scopeId": "test-project"
          }
        ]
      }
    ],
    "enable": true,
    "fullSync": true,
    "isManual": true,
    "labels": [
      "string"
    ],
    "mode": "NORMAL",
    "name": "string",
    "plan": [
      [
        {}
      ]
    ],
    "projectName": "test-project",
    "skipCollectors": true,
    "skipOnFail": true
  },
  "description": "string",
  "enable": true,
  "metrics": [
    {
      "enable": true,
      "pluginName": "string",
      "pluginOption": [
        0
      ]
    }
  ],
  "name": "test"
}'

And then the project named 'test' was created. I tried to create Blueprint by this command,

curl -X 'POST' \
  'https://xxxxx/api/blueprints' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "connections": [
    {
      "connectionId": 5,
      "pluginName": "gitextractor",
      "scopes": [
        {
          "scopeId": "test-project-1"
        },
    {
          "scopeId": "test-project-2"
        },
    {
          "scopeId": "test-project-3"
        }
      ]
    }
  ],
  "cronConfig": "0 0 * * *",
  "enable": true,
  "fullSync": true,
  "isManual": true,
  "labels": [],
  "mode": "NORMAL",
  "name": "test-Blueprint",
  "plan": [
    [
      {}
    ]
  ],
  "projectName": "test",
  "skipCollectors": true,
  "skipOnFail": true
}'

The error message indicates that when specifying "gitextractor" as the plugin for creating a Blueprint, an issue occurred: {"code":0,"success":false,"message":"error creating blueprint (400)\nWraps: (2)\nWraps: (3) plugin gitextractor does not support DataSourcePluginBlueprintV200\nError types: (1) *hintdetail.withDetail (2) *hintdetail.withDetail (3) *errors.errorString","causes":["plugin gitextractor does not support DataSourcePluginBlueprintV200"],"data":null} What should I specify for the Plugin?

[jensrotne]

@kamiyariko I haven't used gitlab before, but I'm pretty sure that the connection plugin name you put in is incorrect. I think it might needs to be Gitlab and not Gitextractor.

Additionally, nothing will happen before you put in the Blueprint plans.

[kamiyariko]

@jensrotne I changed the PluginName to "gitlab" and executed the command, but I encountered this error message. {"code":0,"success":false,"message":"error creating blueprint (400)\nWraps: (2)\nWraps: (3) *models.GitlabProject not found (404)\nWraps: (4) record not found (404)\nWraps: (5) record not found\nError types: (1) *hintdetail.withDetail (2) *hintdetail.withDetail (3) *hintdetail.withDetail (4) *hintdetail.withDetail (5) *errors.errorString","causes":["*models.GitlabProject not found (404)","record not found (404)"],"data":null} If I have already assigned existing scopeIds to the Connection, is there anything else I need to do?

Should I write something in afterPlan and beforePlan? I just want to attach the Blueprint to projects that collect data from GitLab.

[jensrotne]

@kamiyariko one thing you could try is to create the Project and connections manually in the config-ui, and then use the GET /api/blueprints/{blueprint_id} endpoint to see how it should look.

That's at least what I did when I made my API integration.

[klesh]

You need to create the GitlabProject by calling the PutScopes API beforehand

[klesh]

I suggest that you use the config-ui to go through the whole setup process and observe what APIs were called.

[kamiyariko]

I tried the PutScopes API but it was just associating scope with scope-config.

curl -X 'PUT' \
  'https://xxxxx/api/plugins/gitlab/connections/5/scopes' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "data": [
    {
      "archived": true,
      "connectionId": 5,
      "creatorId": 0,
      "defaultBranch": "string",
      "description": "string",
      "forkedFromProjectId": 0,
      "forkedFromProjectWebUrl": "string",
      "gitlabId": 3,
      "httpUrlToRepo": "string",
      "name": "string",
      "openIssuesCount": 0,
      "pathWithNamespace": "string",
      "scopeConfigId": 4,
      "starCount": 0,
      "visibility": "string",
      "webUrl": "string"
    }
  ]
}'

"Thank you for your support. Since @jensrotne has created a bug ticket, let's continue the discussion there. I will close this question now.