search

magit / forge

Work with Git forges from the comfort of Magit
GNU General Public License v3.0
1.31k stars 116 forks source link

Company GitLab instance cannot be pulled from but creates thousands of ` *http foo.company.lan*-N` buffers #709

Open tsdh opened 1 week ago

tsdh commented 1 week ago

I'm trying to get Forge working with our company's GitLab instance but fail. That instance uses only http, not https, so I've added the relevant entries to ghub-insecure-hosts.

(use-package forge
  :ensure t
  :after magit
  :custom
  (forge-bug-reference-hooks nil)
  :config
  (setq ghub-insecure-hosts '("srv-upsource.shd.lan/gitlab"
                              "srv-upsource.shd.lan/gitlab/api/v4"))
  (add-to-list 'forge-alist
               `("srv-upsource.shd.lan" "srv-upsource.shd.lan/gitlab/api/v4"
                 "srv-upsource.shd.lan/gitlab" forge-gitlab-repository)))

Now when I open a Magit status buffer for a project hosted there and do ' f f a to pull all topics, I see the mode line shows "Pulling group/project" but nothing seems to happen. Except that Emacs seems to get slower after some time.

By accident I discovered that I have more than 5000 buffers named *http srv-upsource.shd.lan*. Well, that's the name of the first and the others have a -N suffix with N being an integer. It seems that I get about 5-10 more such buffers every second. They all have the same contents except for the timestamps, e.g., the JSON is always the same. Buffer contents of *http srv-upsource.shd.lan*-127110

I edebugged url-retrieve to get a backtrace:

  url-retrieve(#s(url :type "http" :user nil :password nil :host "srv-upsource.shd.lan" :portspec nil :filename "/gitlab/api/v4/projects/ecoro%2Fecoro" :target nil :attributes nil :fullness t :silent nil :use-cookies t :asynchronous t) ghub--handle-response (#s(ghub--req :url #s(url :type "http" :user nil :password nil :host "srv-upsource.shd.lan" :portspec nil :filename "/gitlab/api/v4/projects/ecoro%2Fecoro" :target nil :attributes nil :fullness t :silent nil :use-cookies t :asynchronous t) :forge gitlab :silent nil :method "GET" :headers #f(compiled-function () #<bytecode -0x14a0d29e270f865a>) :handler ghub--handle-response :unpaginate nil :noerror t :reader nil :buffer #<buffer magit: ecoro-master> :callback #f(compiled-function (value headers status req) #<bytecode -0x183bb40164496ccb>) :errorback t :value nil :extra nil)) nil)
  ghub--retrieve(nil #s(ghub--req :url #s(url :type "http" :user nil :password nil :host "srv-upsource.shd.lan" :portspec nil :filename "/gitlab/api/v4/projects/ecoro%2Fecoro" :target nil :attributes nil :fullness t :silent nil :use-cookies t :asynchronous t) :forge gitlab :silent nil :method "GET" :headers #f(compiled-function () #<bytecode -0x14a0d29e270f865a>) :handler ghub--handle-response :unpaginate nil :noerror t :reader nil :buffer #<buffer magit: ecoro-master> :callback #f(compiled-function (value headers status req) #<bytecode -0x183bb40164496ccb>) :errorback t :value nil :extra nil))
  ghub-request("GET" "/projects/ecoro%2Fecoro" nil :forge gitlab :query nil :payload nil :headers nil :silent nil :unpaginate nil :noerror nil :reader nil :username nil :auth forge :host "srv-upsource.shd.lan/gitlab/api/v4" :callback #f(compiled-function (value headers status req) #<bytecode -0x183bb40164496ccb>) :errorback t :extra nil)
  glab-get("/projects/ecoro%2Fecoro" nil :host "srv-upsource.shd.lan/gitlab/api/v4" :auth forge :query nil :payload nil :headers nil :silent nil :unpaginate nil :noerror nil :reader nil :callback #f(compiled-function (value headers status req) #<bytecode -0x183bb40164496ccb>) :errorback t)
  forge--glab-get(#<forge-gitlab-repository forge-gitlab-repository-1e206fa00a45> "/projects/:project" nil :callback #f(compiled-function (value headers status req) #<bytecode -0x183bb40164496ccb>))
  #f(compiled-function (repo callback) #<bytecode 0x56a4a6051267efa>)(#<forge-gitlab-repository forge-gitlab-repository-1e206fa00a45> #f(compiled-function (cb &optional v) #<bytecode -0xbdd9c6726757e37>))
  apply(#f(compiled-function (repo callback) #<bytecode 0x56a4a6051267efa>) #<forge-gitlab-repository forge-gitlab-repository-1e206fa00a45> #f(compiled-function (cb &optional v) #<bytecode -0xbdd9c6726757e37>))
  forge--fetch-repository(#<forge-gitlab-repository forge-gitlab-repository-1e206fa00a45> #f(compiled-function (cb &optional v) #<bytecode -0xbdd9c6726757e37>))
  #f(compiled-function (cb &optional v) #<bytecode -0xbdd9c6726757e37>)(#f(compiled-function (cb &optional v) #<bytecode -0xbdd9c6726757e37>) nil)
  #f(compiled-function (value headers status req) #<bytecode -0x183bb40164496ccb>)(nil nil nil #s(ghub--req :url #s(url :type nil :user nil :password nil :host nil :portspec nil :filename nil :target nil :attributes nil :fullness nil :silent nil :use-cookies t :asynchronous t) :forge gitlab :silent nil :method "GET" :headers #f(compiled-function () #<bytecode -0x14a0d29e270f865a>) :handler ghub--handle-response :unpaginate nil :noerror t :reader nil :buffer #<buffer magit: ecoro-master> :callback #f(compiled-function (value headers status req) #<bytecode -0x183bb40164496ccb>) :errorback t :value nil :extra nil))
  ghub--handle-response(nil #s(ghub--req :url #s(url :type nil :user nil :password nil :host nil :portspec nil :filename nil :target nil :attributes nil :fullness nil :silent nil :use-cookies t :asynchronous t) :forge gitlab :silent nil :method "GET" :headers #f(compiled-function () #<bytecode -0x14a0d29e270f865a>) :handler ghub--handle-response :unpaginate nil :noerror t :reader nil :buffer #<buffer magit: ecoro-master> :callback #f(compiled-function (value headers status req) #<bytecode -0x183bb40164496ccb>) :errorback t :value nil :extra nil))
  url-http-activate-callback()
  url-http-chunked-encoding-after-change-function(591 2232 1641)
  url-http-wait-for-headers-change-function(1 2251 2250)
  url-http-generic-filter(#<process srv-upsource.shd.lan> "HTTP/1.1 200 OK\15\nServer: nginx\15\nDate: Tue, 08 Oct 2024 11:51:47 GMT\15\nContent-Type: application/json\15\nTransfer-Encoding: chunked\15\nConnection: keep-alive\15\nVary: Accept-Encoding\15\nCache-Control: max-age=0, private, must-revalidate\15\nEtag: W/\"bebb7252cb612c57a7223c3ed9ede96a\"\15\nVary: Origin\15\nX-Content-Type-Options: nosniff\15\nX-Frame-Options: SAMEORIGIN\15\nX-Gitlab-Meta: {\"correlation_id\":\"01J9NZH6AXHXZP3M8CCY2ADHRB\",\"version\":\"1\"}\15\nX-Request-Id: 01J9NZH6AXHXZP3M8CCY2ADHRB\15\nX-Runtime: 0.088452\15\nStrict-Transport-Security: max-age=63072000\15\nReferrer-Policy: strict-origin-when-cross-origin\15\nContent-Encoding: gzip\15\n...")

The problem seems to be that the &optional v arg of the callback cb lambda in forge--pull for gitlab is always nil. Which in turn seems to be because the :callback lambda in forge--fetch-repository doesn't assign to value because neither does (oref repo selective-p) hold nor (magit-get-boolean "forge.omitExpensive") and value is and stays nil. Therefore, we call forge--fetch-repository over and over again.

I've tried adding a default case to the cond which sets value to (t). It seems like forge then goes on fetching assignees, forks, and labels but not issues and pull-requests... Hm, I guess instead of t the alist representation of the JSON in *http srv-upsource.shd.lan*-127110 buffer should be added to value. But how?

I use an up-to-date Emacs from the master branch and the current MELPA versions of all magit-related packages:

drwxr-xr-x 2 hta hta 4096 10. Okt 13:28 forge-20241001.2058/
drwxr-xr-x 2 hta hta 4096  8. Okt 13:39 ghub-20241001.1025/
drwxr-xr-x 2 hta hta 4096  8. Okt 13:38 magit-20241004.917/
drwxr-xr-x 2 hta hta 4096  8. Okt 13:38 magit-section-20241001.2052/
tarsius commented 1 week ago

I'm in need of a break but will try to look at this (probably late) next week.

tsdh commented 1 week ago

Hey Jonas, no problem at all. Take as much time as you need.

tsdh commented 1 week ago

Oh, I found the culprit myself in a new debugging session! It's actually in ghub and not forge. I added a small fix locally right at the start of the ghub--handle-response function which makes pulling work for me and our company's GitLab server. But as stated in the FIXME comment, I'm not sure why that has ever worked...

(cl-defun ghub--handle-response (status req &optional (buffer nil sbuf))
  ;; FIXME START: status being nil won't set buf and will make
  ;; ghub--handle-response-error assume that a timeout has happened.
  ;;
  ;; ghub--handle-response is used as callback function of url-retrieve.  Its
  ;; docstring states that the first arg STATUS is a plist with :redirect and
  ;; :error keys or an empty list, if no events have occured which means the
  ;; request worked normally, e.g. no error and no redirects.  That has been
  ;; the case since at least Emacs 27, so I don't understand how this code has
  ;; ever worked.  Maybe there is always a :redirect with https connections (my
  ;; companies GitLab is accessed with plain http because it's only reachable
  ;; interally)?  Who knows...
  (unless status
    (setq status '(:status 200))) ;; Anything non-nil will do.
  ;; FIXME END
  ...
tarsius commented 1 week ago

I don't understand how this code has ever worked

Before the misguided but fairly recent https://github.com/magit/ghub/commit/03d7161cadabce8e371a98f30edf4ad092a72bbd, it didn't do some of the things you are describing.

What was good about that commit is that it tries to gracefully handle (well at least signal) a timeout (and correctly assumes that url.el fails to properly document how callers are informed about timeouts). However "status = nil => timeout" was clearly the wrong conclusion. "return value = nil => timeout" appears to be the correct way to detect it.

To minimize the risk of another broken "fix", I'll take a few days for this.

LuciusChen commented 1 week ago

I cannot add my self-hosted GitLab repository using Forge. The request keeps pending and slows down Emacs. It seems to be related to this issue.

tsdh commented 6 days ago

@LuciusChen Do you also get tons of http buffers? If so, that's probably the same issue. You can check by typing C-x b, then entering a space (the buffers are "hidden"), an asterisk, http, and then TAB to get a list of completions.

LuciusChen commented 6 days ago

@LuciusChen Do you also get tons of http buffers? If so, that's probably the same issue. You can check by typing C-x b, then entering a space (the buffers are "hidden"), an asterisk, http, and then TAB to get a list of completions.

Whenever I add multiple self-hosted GitLab repositories, there are multiple HTTP requests ongoing. When the number reaches a certain level, Emacs becomes very slow and laggy.

tsdh commented 5 days ago

@LuciusChen Alright, then it's probably the same issue.