GitLab Integration Support for Self-Hosted GitLab #418

Open dangtony98 opened 1 year ago

Feature description

Currently, Infisical has 2 integration options for GitLab documented further here:

Standard (direct connection from Infisical to GitLab Cloud via their API).
Pipeline (CLI in GitLab pipeline pulls secrets from Infisical; works for GitLab Cloud and Self-Hosted).

The Standard option only works for sending secrets to GitLab Cloud but users may be self-hosting GitLab on their own infrastructure.

What's needed is a modification to the Standard option to be able to sync secrets from Infisical to GitLab Self-Hosted.

Why would it be useful?

Users self-hosting GitLab would appreciate being able to use the Standard option.

Additional context

This may or may not be complicated because you have to account for:

Authentication with GitLab Self-Hosted (the one currently for GitLab Cloud authenticates via OAuth2 but not sure how it is structured for the self-hosted option).
Record URL/pointer to your GitLab Self-Hosted instance.

Hello @maidul98

I'm following up on this issue for modifying the Standard option to support syncing secrets from Infisical to GitLab Self-Hosted. Are there any updates on this feature? Will it be included in your upcoming roadmap?

Thanks in advance for your response.

Support here, do we have any idea about when this feature will be release? Thanks

Will look into this.

This is already released, we have been using it for some time already. https://infisical.com/docs/integrations/cicd/gitlab - see the Self-Hosted Setup tab

@WladyX I believe, we need a way to add Gitlab Self-Hosted URL. As there is no way to config. "Self-Hosted Setup tab" is for Infisical Self-Hosted setup.

I have both, Infisical selfhosted and gitlab selfhosted. When you go to integrations in the infisical UI under gitlab you can add the selfhosted url, like in the screenshot: CleanShot 2024-02-29 at 17 15 12@2x

I have both, Infisical selfhosted and gitlab selfhosted. When you go to integrations in the infisical UI under gitlab you can add the selfhosted url, like in the screenshot:

i have the same form too, but when i try to connect with my Gitlab CE tenant:

Back in your Infisical instance, add two new environment variables for the credentials of your GitLab application:

CLIENT_ID_GITLAB: The Client ID of your GitLab application.
CLIENT_SECRET_GITLAB: The Secret of your GitLab application.
Once added, restart your Infisical instance and use the GitLab integration.

Have you done this? As per the docs?

Back in your Infisical instance, add two new environment variables for the credentials of your GitLab application:

CLIENT_ID_GITLAB: The Client ID of your GitLab application.
CLIENT_SECRET_GITLAB: The Secret of your GitLab application.
Once added, restart your Infisical instance and use the GitLab integration.

Have you done this? As per the docs?

That's required only in Self hosted instance of Infisical Using the GitLab integration on a self-hosted instance of Infisical requires configuring an application in GitLab and registering your instance with it.

I am on Infisical Cloud + GitLab CE

It make sense, why this is not working? The self-hosted URL and the CLIENT_ID/CLIENT_SECRET in the Infisical cloud are not the same. Since this is issue related only to Infisical cloud. I think someone from the core team, should take a look.

@maidul98 can we mark this as a real issue / bug? what do you think? Thanks

Hi folks!

I've just checked the flow and I believe that Infisical Cloud is currently configured specifically to sync to GitLab Cloud.

That said, you can sync to a self-hosted instance of GitLab if you self-host Infisical yourself and configure it appropriately with the right CLIENT_ID_GITLAB and CLIENT_SECRET_GITLAB as mentioned by @WladyX . We'll definitely look into adding support for syncing from Infisical Cloud to self-hosted GitLab instances at some point soon but in the meantime do feel free to send in PRs for it.

Alternatively, for any enterprises, we'd be happy to provide a dedicated, managed Infisical Cloud instance that is able to sync to your self-hosted GitLab instance. If this is of interest, feel free to drop a line to team@infisical.com.

Hi folks!

I've just checked the flow and I believe that Infisical Cloud is currently configured specifically to sync to GitLab Cloud.

That said, you can sync to a self-hosted instance of GitLab if you self-host Infisical yourself and configure it appropriately with the right CLIENT_ID_GITLAB and CLIENT_SECRET_GITLAB as mentioned by @WladyX . We'll definitely look into adding support for syncing from Infisical Cloud to self-hosted GitLab instances at some point soon but in the meantime do feel free to send in PRs for it.

Alternatively, for any enterprises, we'd be happy to provide a dedicated, managed Infisical Cloud instance that is able to sync to your self-hosted GitLab instance. If this is of interest, feel free to drop a line to team@infisical.com.

Ok thanks @dangtony98 for the quick response.

Hi, May be this is the wrong thread but does infisical support glabally available application on gitlab self-hosted instance ? I've created an application from the admin area on gitlab and set the CLIENT_ID_GITLAB and CLIENT_SECRET_GITLAB accordingly. I've also set the GITLAB_URL as an environment variable even if it will be set from the UI. When I try to complete the gitlab integration after setting GitLab Integration Type to Individual, there are no projects in the list (no projects found). On the logs side, I can see this error:

{"level":50,"time":1709639613858,"pid":1,"hostname":"c43430d439b4","reqId":"req-3o","severity":"ERROR","err":{"message":"Request failed with status code 404","name":"AxiosError","stack":"AxiosError: Request failed with status code 404\n    at settle (file:///backend/node_modules/axios/lib/core/settle.js:19:12)\n    at IncomingMessage.handleStreamEnd (file:///backend/node_modules/axios/lib/adapters/http.js:589:11)\n    at IncomingMessage.emit (node:events:530:35)\n    at IncomingMessage.emit (node:domain:488:12)\n    at endReadableNT (node:internal/streams/readable:1696:12)\n    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)\n    at Axios.request (file:///backend/node_modules/axios/lib/core/Axios.js:45:41)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async getAppsGitlab (file:///backend/dist/services/integration-auth/integration-app-list.mjs:287:24)\n    at async Object.getIntegrationApps (file:///backend/dist/services/integration-auth/integration-auth-service.mjs:212:18)\n    at async Object.handler (file:///backend/dist/server/routes/v1/integration-auth-router.mjs:258:20)","config":{"transitional":{"silentJSONParsing":true,"forcedJSONParsing":true,"clarifyTimeoutError":false},"adapter":["xhr","http"],"transformRequest":[null],"transformResponse":[null],"timeout":0,"xsrfCookieName":"XSRF-TOKEN","xsrfHeaderName":"X-XSRF-TOKEN","maxContentLength":-1,"maxBodyLength":-1,"env":{},"headers":{"Accept":"application/json, text/plain, */*","Authorization":"Bearer e32c0ec286e1b79a17da437f075a252cfc492df25cbe7ffaa3306e60478277c0","Accept-Encoding":"application/json","User-Agent":"axios/1.6.7"},"params":{},"method":"get","url":"https://gitlab.company.com/api/v4/groups/undefined/projects","axios-retry":{"retries":3,"shouldResetTimeout":false,"retryCount":0,"lastRequestTime":1709639613801}},"code":"ERR_BAD_REQUEST","status":404},"msg":"Request failed with status code 404"}

if I set Group instead of Individual the application, It returns 500:

Something went wrong. Please contact [support@infisical.com](mailto:support@infisical.com) if the issue persists.

and on the logs I have this error:

{"level":50,"time":1709639614918,"pid":1,"hostname":"c43430d439b4","reqId":"req-3p","severity":"ERROR","req":{"method":"GET","url":"/api/v1/integration-auth/8c6ea3fe-b4a6-400a-9992-dfdd3bd2d46b/apps?teamId=undefined","hostname":"infisical.company.com","remoteAddress":"161.106.88.17","remotePort":44210},"res":{"statusCode":500},"err":{"message":"Request failed with status code 404","name":"AxiosError","stack":"AxiosError: Request failed with status code 404\n    at settle (file:///backend/node_modules/axios/lib/core/settle.js:19:12)\n    at IncomingMessage.handleStreamEnd (file:///backend/node_modules/axios/lib/adapters/http.js:589:11)\n    at IncomingMessage.emit (node:events:530:35)\n    at IncomingMessage.emit (node:domain:488:12)\n    at endReadableNT (node:internal/streams/readable:1696:12)\n    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)\n    at Axios.request (file:///backend/node_modules/axios/lib/core/Axios.js:45:41)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async getAppsGitlab (file:///backend/dist/services/integration-auth/integration-app-list.mjs:287:24)\n    at async Object.getIntegrationApps (file:///backend/dist/services/integration-auth/integration-auth-service.mjs:212:18)\n    at async Object.handler (file:///backend/dist/server/routes/v1/integration-auth-router.mjs:258:20)","config":{"transitional":{"silentJSONParsing":true,"forcedJSONParsing":true,"clarifyTimeoutError":false},"adapter":["xhr","http"],"transformRequest":[null],"transformResponse":[null],"timeout":0,"xsrfCookieName":"XSRF-TOKEN","xsrfHeaderName":"X-XSRF-TOKEN","maxContentLength":-1,"maxBodyLength":-1,"env":{},"headers":{"Accept":"application/json, text/plain, */*","Authorization":"Bearer e32c0ec286e1b79a17da437f075a252cfc492df25cbe7ffaa3306e60478277c0","Accept-Encoding":"application/json","User-Agent":"axios/1.6.7"},"params":{},"method":"get","url":"https://gitlab.company.com/api/v4/groups/undefined/projects","axios-retry":{"retries":3,"shouldResetTimeout":false,"retryCount":0,"lastRequestTime":1709639614890}},"code":"ERR_BAD_REQUEST","status":404},"msg":"Request failed with status code 404"}
{"level":30,"time":1709639614918,"pid":1,"hostname":"c43430d439b4","reqId":"req-3p","severity":"INFO","res":{"statusCode":500},"responseTime":34.870799999684095,"msg":"request completed"}

Would it be possible to have a unique application instead of registering a per-group or a per-project application ?

Note: I'm using version v0.46.5-postgres.

Thank you.

Encountered the same issue as mentioned by @atefhaloui and the likely culprit is that the backend expects IDs returned from the instance to be Strings but they are integer

https://github.com/Infisical/infisical/blob/635948c4f43f855e344ba0fcc0fb7b5c89ee70d4/backend/src/services/integration-auth/integration-team.ts#L25

I checked and my instance definitely returns an integer (1, 2, 3 ...) for project ID.

And further down the stack there is Zod validation on teamId being a string and since its optional I assume it just drops it as its missing in the response the frontend gets

@dangtony98 is PR welcome or do you want to fix this yourself?

Actually now I see it - the integration-team puts it as teamId but the Zod schema as id - that's why its not included and no validation error so the bugs are two - the type and the zod schema which should be teamId: z.string().optional() (maybe remove the optional?)

EDIT:

tested and it works, changing t.id to String(t.id) and id -> teamId in the Zod schema

I am also having this error and created a new issue (https://github.com/Infisical/infisical/issues/1875#issue-2317881687) to make sure this is seen as a distinct bug.

Infisical / infisical