search

fsprojects / Paket

A dependency manager for .NET with support for NuGet packages and Git repositories.
https://fsprojects.github.io/Paket/
MIT License
2.02k stars 520 forks source link

Downloading github files through proxy #1832

Open JohanLarsson opened 8 years ago

JohanLarsson commented 8 years ago

Description

Paket does not work with github files from behind proxy.

When building paket.sln:

Severity    Code    Description Project File    Line    Suppression State
Error       Could not download from https://raw.githubusercontent.com/fsharp/FAKE/7f9fe8f546f6adec66febeec31e50821b60814d6/modules/Octokit/Octokit.fsx
 Message: One or more errors occurred.
 Details: The underlying connection was closed: An unexpected error occurred on a send.     C:\Git\Third Party\Paket\integrationtests\Paket.IntegrationTests\paket.references   0

Repro steps

#r "System.dll"
open System.Net

let https = @"https://raw.githubusercontent.com/fsharp/FAKE/f4024fd5b790485828c1cfc002190716eee97597/modules/Octokit/Octokit.fsx";
let http = @"http://stackoverflow.com/"

let download (url:string) =
    printfn "%s" url;
    try
      use client = new WebClient()
      // client.Proxy.Credentials <- System.Net.CredentialCache.DefaultNetworkCredentials
      client.DownloadString url |> ignore
      printfn "Success!"
    with
    | ex -> printfn "Exception! %s " (ex.Message)

download http
download https

Uncommenting the proxy line gives success for http but exception for https.

Exception! The remote server returned an error: (403) Forbidden.

I don't know much about proxies, been testing a bunch of stackoverflow answers but nothing has worked. If anyone has suggestions please paste a modified script and I'll run it and output the results.

Known workarounds

a) Download the files manually and turn off auto-restore. b) This is a nasty hack, but it works for downloading the file. It is Windows only and brings in WPF or WinForms references which is not ideal.

Related information

forki commented 8 years ago

/cc @vasily-kirichenko you are behind proxy right? Do paket's github deps work for you?

forki commented 8 years ago

https://github.com/fsprojects/Paket/blob/master/src/Paket.Core/Utils.fs#L355 is how we handle proxies.

@JohanLarsson that one gets called during github download, right?

vasily-kirichenko commented 8 years ago

@forki I've never used GitHub deps.

vilinski commented 8 years ago

should be like here:

  use client = new WebClient()
  let proxy = WebRequest.DefaultWebProxy
  proxy.Credentials <- CredentialCache.DefaultCredentials
  client.Proxy <- proxy
vasily-kirichenko commented 8 years ago

@vilinski no, this does not work for me.

vilinski commented 8 years ago

That's strange, it should work, works for me behind authorized corp firewall. Other method - not to touch code at any places, but set this into app.config:

  <system.net>
      <defaultProxy useDefaultCredentials="true" />
  </system.net>
forki commented 8 years ago

strange thing is all other http calls seem to work - @JohanLarsson are we really calling https://github.com/fsprojects/Paket/blob/master/src/Paket.Core/Utils.fs#L355 for github deps?

JohanLarsson commented 8 years ago

@forki I'll change to client.Proxy <- getDefaultProxyFor url in the script and test it, need to go to work to do it, hopefully tonight. Regarding if it is called I'll try the manual download workaround and set a bp. Reporting back as soon as I have results.

@vilinski I'll try that permutation, thanks, pretty sure I have tried it before.

@vasily-kirichenko Can you build paket? It fails at work for me on trying to restore Octokit.fsx.

The proxy is causing pain for many other apps, think it might be a nasty one.

JohanLarsson commented 8 years ago

@forki Confirmed bp gets hit in getDefaultProxyForwhen running install.

forki commented 8 years ago

Yeah but is that still failing? It could still be a normal nuget call (from unrelated code)

On Jul 26, 2016 6:13 PM, "Johan Larsson" notifications@github.com wrote:

@forki https://github.com/forki Confirmed bp gets hit in getDefaultProxyForwhen running install.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/fsprojects/Paket/issues/1832#issuecomment-235319880, or mute the thread https://github.com/notifications/unsubscribe-auth/AADgNHVeNsvzKURvDO2wKyxI-EY9IWWkks5qZjIhgaJpZM4JUUfj .

JohanLarsson commented 8 years ago

It fails but the stack does not reveal much so I can't tell for sure yet if it got called in the failing path. I tried using a paket.dependencies with only a github ref but then I ended up here

Edit: image

Looks like it gets called.

JohanLarsson commented 8 years ago

Confirmed did not work: https://github.com/fsprojects/Paket/issues/1832#issuecomment-235186799

nojaf commented 5 years ago

I'm having something strange here. paket.dependencies:

source https://www.nuget.org/api/v2
storage:none

clitool dotnet-fable
nuget Fable.Core
nuget Fable.Import.Node
nuget Fable.Import.Browser
nuget Fable.Import.Jest
nuget Fable.PowerPack

group Build
  source https://nuget.org/api/v2
  nuget FAKE
  github fable-compiler/fake-helpers Fable.FakeHelpers.fs

Paket 5.181.1 cannot download files from github

C:\temp\fable-import>.paket\paket.exe install
Paket version 5.181.1
Skipping resolver for group Build since it is already up-to-date
Skipping resolver for group Main since it is already up-to-date
C:\temp\fable-import\paket.lock is already up-to-date
Installing into projects:
 - Creating model and downloading packages.
Downloading fable-compiler/fake-helpers:46b48de9f1b532b63b015a9e2e2b22d150734728 Fable.FakeHelpers.fs to C:\temp\fable-import\paket-files\build\fable-compiler\fake-helpers\Fable.FakeHelpers.fs
  - Download failed. Trying again. (4 trials left)

While if I switch to dotnet core local tool

dotnet tool install Paket --version 5.182.0-alpha001 --tool-path ".paket" --add-source https://api.nuget.org/v3/index.json

I can download the files from github.

PS C:\temp\fable-import> dotnet tool install  Paket --version 5.182.0-alpha001 --tool-path ".paket" --add-source https://api.nuget.org/v3/index.json
You can invoke the tool using the following command: paket
Tool 'paket' (version '5.182.0-alpha001') was successfully installed.
PS C:\temp\fable-import> paket install

C:\temp\fable-import>.paket\paket.exe install
Paket version 5.182.0-alpha001
Skipping resolver for group Build since it is already up-to-date
Skipping resolver for group Main since it is already up-to-date
C:\temp\fable-import\paket.lock is already up-to-date
Installing into projects:
 - Creating model and downloading packages.
 - Animejs/paket.references -> Animejs/Fable.Import.Animejs.fsproj
 - Browser/paket.references -> Browser/Fable.Import.Browser.fsproj
 - Electron/paket.references -> Electron/Fable.Import.Electron.fsproj
 - Express/paket.references -> Express/Fable.Import.Express.fsproj
 - GeoJSON/paket.references -> GeoJSON/Fable.Geojson.fsproj
 - GoogleMaps/paket.references -> GoogleMaps/Fable.Import.GoogleMaps.fsproj
 - HMR/paket.references -> HMR/Fable.Import.HMR.fsproj
 - Leaflet/paket.references -> Leaflet/Fable.Leaflet.fsproj
 - Mocha/paket.references -> Mocha/Fable.Import.Mocha.fsproj
 - Node/paket.references -> Node/Fable.Import.Node.fsproj
 - Pg/paket.references -> Pg/Fable.Import.Pg.fsproj
 - Pixi/paket.references -> Pixi/Fable.Import.Pixi.fsproj
 - PixiParticles/paket.references -> PixiParticles/Fable.Import.Pixi.Particles.fsproj
 - PixiSound/paket.references -> PixiSound/Fable.Import.Pixi.Sound.fsproj
 - SocketIo/paket.references -> SocketIo/Fable.Import.SocketIo.fsproj
 - test/paket.references -> test/Base.fsproj
 - test/Node/paket.references -> test/Node/NodeTest.fsproj
 - Uws/paket.references -> Uws/Fable.Import.Uws.fsproj
 - VSCode/paket.references -> VSCode/Fable.Import.VSCode.fsproj
Performance:
 - Disk IO: 28 milliseconds
 - Runtime: 2 seconds
PS C:\temp\fable-import> Test-Path .\paket-files\build\fable-compiler\fake-helpers\Fable.FakeHelpers.fs
True

/cc @enricosada