search

datalad / datalad

Keep code, data, containers under control with git and git-annex
http://datalad.org
Other
540 stars 111 forks source link

`datalad get` throw error `download failed: Network.Socket.recvBuf: invalid argument (Invalid argument)` #7676

Open NeroBlackstone opened 1 day ago

NeroBlackstone commented 1 day ago

Description

Hi, I tried to complete this part of the tutorial

I'm using Windows 11 and git bash, datalad v1.1.3, when I execute:

datalad get Long_Now__Seminars_About_Long_term_Thinking/2003_11_15__Brian_Eno__The_Long_Now.mp3

I got a error:

get(error): Long_Now__Seminars_About_Long_term_Thinking\2003_11_15__Brian_Eno__The_Long_Now.mp3 (file) [http://podcast.longnow.org/salt/redirect/salt-020031114-eno-podcast.mp3 download failed: Network.Socket.recvBuf: invalid argument (Invalid argument)
downloading from all 1 known url(s) failed
http://podcast.longnow.org/salt/redirect/salt-020031114-eno-podcast.mp3 download failed: Network.Socket.recvBuf: invalid argument (Invalid argument)
downloading from all 1 known url(s) failed
http://podcast.longnow.org/salt/redirect/salt-020031114-eno-podcast.mp3 download failed: Network.Socket.recvBuf: invalid argument (Invalid argument)
downloading from all 1 known url(s) failed]

And if I ping podcast.longnow.org:

$ ping podcast.longnow.org

Pinging podcast.longnow.org [13.33.21.87] with 32 bytes of data:
Reply from 13.33.21.87: bytes=32 time=187ms TTL=245
Reply from 13.33.21.87: bytes=32 time=190ms TTL=245
Reply from 13.33.21.87: bytes=32 time=187ms TTL=245
Reply from 13.33.21.87: bytes=32 time=186ms TTL=245

Ping statistics for 13.33.21.87:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 186ms, Maximum = 190ms, Average = 187ms

I can access this mp3 via broswer: image

But if I download through git annex, it also download faild:

$  git annex get -d -v  Long_Now__Seminars_About_Long_term_Thinking/2003_11_15__Brian_Eno__The_Long_Now.mp3
[2024-10-31 14:52:20.5061622] (Utility.Process) process [4748] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","ls-files","--stage","-z","--error-unmatch","--","Long_Now__Seminars_About_Long_term_Thinking/2003_11_15__Brian_Eno__The_Long_Now.mp3"]
[2024-10-31 14:52:20.508717] (Utility.Process) process [19548] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)","--buffer"]
[2024-10-31 14:52:20.5113637] (Utility.Process) process [6236] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","cat-file","--batch=%(objectname) %(objecttype) %(objectsize)","--buffer"]
[2024-10-31 14:52:20.5144701] (Utility.Process) process [18928] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","git-annex"]
[2024-10-31 14:52:20.5434736] (Utility.Process) process [18928] done ExitSuccess
[2024-10-31 14:52:20.5454882] (Utility.Process) process [11148] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","show-ref","--hash","refs/heads/git-annex"]
[2024-10-31 14:52:20.5640457] (Utility.Process) process [11148] done ExitSuccess
[2024-10-31 14:52:20.5673022] (Utility.Process) process [19920] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","log","refs/heads/git-annex..0eb95b03a1de860c70f0351962201f2df897a14d","--pretty=%H","-n1"]
[2024-10-31 14:52:20.587733] (Utility.Process) process [19920] done ExitSuccess
[2024-10-31 14:52:20.5933695] (Utility.Process) process [14632] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","hash-object","-w","--no-filters","--stdin-paths"]
[2024-10-31 14:52:20.5958201] (Utility.Process) process [12312] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","cat-file","--batch"]
[2024-10-31 14:52:20.5991369] (Utility.Process) process [5320] feed: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","update-index","-z","--index-info"]
[2024-10-31 14:52:20.6022596] (Utility.Process) process [18892] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","diff-index","--raw","-z","-r","--no-renames","-l0","--cached","refs/heads/git-annex","--"]
[2024-10-31 14:52:20.631118] (Utility.Process) process [18892] done ExitSuccess
[2024-10-31 14:52:20.6334821] (Utility.Process) process [5320] done ExitSuccess
[2024-10-31 14:52:20.6481634] (Utility.Process) process [19304] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","-c","annex.debug=true","cat-file","--batch=%(objectname) %(objecttype) %(objectsize)","--buffer"]
get Long_Now__Seminars_About_Long_term_Thinking/2003_11_15__Brian_Eno__The_Long_Now.mp3 (from web...)
[2024-10-31 14:52:20.6957438] (Utility.Url) Request {
  host                 = "podcast.longnow.org"
  port                 = 80
  secure               = False
  requestHeaders       = [("Accept-Encoding","identity"),("User-Agent","git-annex/10.20240927-gfca26db22b8ad128c542c30b9d2983ec72dbe16b")]
  path                 = "/salt/redirect/salt-020031114-eno-podcast.mp3"
  queryString          = ""
  method               = "GET"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
  proxySecureMode      = ProxySecureWithConnect
}

  http://podcast.longnow.org/salt/redirect/salt-020031114-eno-podcast.mp3 download failed: Network.Socket.recvBuf: invalid argument (Invalid argument)

  downloading from all 1 known url(s) failed

  Unable to access these remotes: web

  Maybe add some of these git remotes (git remote add ...):
        da3bf937-5bd2-43ea-a07b-bcbe71f3b875 -- mih@medusa:/tmp/seminars-on-longterm-thinking

  (Note that these git remotes have annex-ignore set: origin)
failed
[2024-10-31 14:52:21.2573074] (Utility.Process) process [19304] done ExitSuccess
[2024-10-31 14:52:21.2573074] (Utility.Process) process [6236] done ExitSuccess
[2024-10-31 14:52:21.2583934] (Utility.Process) process [19548] done ExitSuccess
[2024-10-31 14:52:21.2583934] (Utility.Process) process [4748] done ExitSuccess
[2024-10-31 14:52:21.2616549] (Utility.Process) process [12312] done ExitSuccess
[2024-10-31 14:52:21.2646848] (Utility.Process) process [14632] done ExitSuccess
get: 1 failed

I don't know what happened, but I think ·Network.Socket.recvBuf: invalid argument (Invalid argument)· is the key to the problem.

NeroBlackstone commented 1 day ago

I know the reason, http://podcast.longnow.org/salt/redirect/salt-020031114-eno-podcast.mp3 will redirect to http://dts.podtrac.com/redirect.mp3/media.blubrry.com/longnow/podcast.longnow.org/salt/salt-020031114-eno-podcast.mp3， and dts.podtrac.com block in China.

Close issue, this is a network issue, I will config http proxy and try to solve this problem.

NeroBlackstone commented 1 day ago

After config a proxy, I can curl salt-020031114-eno-podcast.mp3 and got a 200, but datalad get still failed... Do I need to set up a separate http proxy for git annex？

PS C:\Users\TY\Documents\DataLad-101\recordings\longnow> curl http://podcast.longnow.org/salt/redirect/salt-020031114-eno-podcast.mp3

StatusCode        : 200
StatusDescription : OK
Content           : {73, 68, 51, 3...}
RawContent        : HTTP/1.1 200 OK
                    Connection: close
                    Age: 1637
                    X-Amz-Cf-Id: SGLh7ddZSkdQ0n9XTiYBzx-eJQVGnRQEUVHXX4ovHhUjxlsboI17TA==
                    X-Amz-Cf-Pop: HKG54-P2
                    X-Cache: Hit from cloudfront
                    Content-Length: 37400577
                    Co...
Headers           : {[Connection, close], [Age, 1637], [X-Amz-Cf-Id,
                    SGLh7ddZSkdQ0n9XTiYBzx-eJQVGnRQEUVHXX4ovHhUjxlsboI17TA==], [X-Amz-Cf-Pop, HKG54-P2]...}
RawContentLength  : 37400577

PS C:\Users\TY\Documents\DataLad-101\recordings\longnow> datalad get Long_Now__Seminars_About_Long_term_Thinking/2003_11_15__Brian_Eno__The_Long_Now.mp3
get(error): Long_Now__Seminars_About_Long_term_Thinking\2003_11_15__Brian_Eno__The_Long_Now.mp3 (file) [http://podcast.longnow.org/salt/redirect/salt-020031114-eno-podcast.mp3 download failed: Network.Socket.recvBuf: invalid argument (Invalid argument)
downloading from all 1 known url(s) failed
http://podcast.longnow.org/salt/redirect/salt-020031114-eno-podcast.mp3 download failed: Network.Socket.recvBuf: invalid argument (Invalid argument)
downloading from all 1 known url(s) failed
http://podcast.longnow.org/salt/redirect/salt-020031114-eno-podcast.mp3 download failed: Network.Socket.recvBuf: invalid argument (Invalid argument)
downloading from all 1 known url(s) failed]
adswa commented 1 day ago

this is difficult for me to test, but you could - just to get datalad out of the equation - try running git annex get instead of datalad get, and if the issue is in git-annex, ask at https://git-annex.branchable.com/bugs for additional insight