Closed krakorj closed 6 years ago
MichaelSimons
commented
6 years ago @krakorj - How do you authenticate to the private NuGet feed when running locally? Where are your credentials being stored/how are they specified? These will need to be setup within the Docker container. It would be similar to setting up a new dev or build machine.
krakorj
commented
6 years ago Do you mean the authentication to the Nuget server via the nuget.config? We do it for TFS, the https://github.com/emgarten/Sleet solution the server is without authentication.
MichaelSimons
commented
6 years ago Yes, how do you authenticate to the NuGet server? Can you explain what you mean by "we do it for TFS"?
krakorj
commented
6 years ago We use command
nuget config -configFile ./nuget.config -set http_proxy=https://proxy.server.com:8080 -set http_proxy.user=example\myname -set http_proxy.password=PWDnuget.config is as follows:
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<packageSources>
<add key="kbciot" value="http://s900b216/SEM.NuGetServer.v3/feed/index.json" protocolVersion="3" />
<add key="nuget.org" value="https://api.nuget.org/v3/index.json" protocolVersion="3" />
</packageSources>
<config>
<add key="http_proxy" value="https://proxy.server.com:8080" />
<add key="http_proxy.user" value="example\myname" />
<add key="http_proxy.password" value="AQAAANCMnd8BFdERjHoAwE/Cl+sBAAAAJayVH7h9MEmfhGubg0dzngAAAAACAAAAAAADZgAAwAAAABAAAABiEH5MLxRtR/BKciw/krndAAAAAASAAACgAAAAEAAAAAmdvBMUqYd152qdnLhyWfcQAAAAO6CxXIcd9V664OoHw/4u5xQAAACqvEj0eupJUNkJTMuTVHgxEK87Pg==" />
</config>
</configuration>@krakorj - You are going to need to do the same within your Dockerfile. You can use the ARG instruction to pass secrets in necessary depending on your scenario.
krakorj
commented
6 years ago @MichaelSimons Can you give me a hint how to use the ARGs? The secrets are already in the nuget.config , we copy the file to workspace directory (in Dockerfile COPY *.config ./). Why shall I use the arguments?
krakorj
commented
6 years ago We found a workaround since the nuget repository is not reachable from the build docker image process:
msbuild *.sln /t:buildmsbuild *.sln /t:publish /p:PublishWithAspNetCoreTargetManifest=falseDockerfile:
FROM microsoft/dotnet:2.0-sdk
WORKDIR /appCOPY /folder/where/published/. ./
ENTRYPOINT ["dotnet", "target.dll"]
@krakorj - I wouldn't consider that a great workaround as you are not getting the full benefits or dockerizing your build. I will create a sample and post it here.
MichaelSimons
commented
6 years ago @krakorj - The underlying cause of the issue you encountered is that the NuGet CLI cmds that set the pwds in the config file encrypt the passwords (e.g. nuget config -configFile ./nuget.config -set http_proxy=https://proxy.server.com:8080 -set http_proxy.user=example\myname -set http_proxy.password=PWD). This makes the resulting config files machine specific. To address this, you would have to run the NuGet cmd inside your Dockerfile/container. Unfortunately there are two problems that make this difficult. The first is that the NuGet CLI (which is based on the .NET Full FX) doesn't support NanoServer. The other problem is that the Nuget config cmd doesn't support a -StorePasswordInClearText option.
I have put included a couple samples that illustrate how to work-around these issue depending on your scenario.
Build cmd: docker build -t test --build-arg MY_PRIVATE_NUGET_SOURCE_PWD=secret_pwd .
# escape=`
# NuGet config image - NuGet CLI has to be run in a WindowsServerCore container, as a result, PWD must be stored in clear text in order to be used in a later stage. The NuGet config won't be in the runtime image so this shouldn't be an issue.
FROM microsoft/windowsservercore:1709 AS nuget-config
SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]
RUN Invoke-WebRequest -OutFile nuget.exe https://dist.nuget.org/win-x86-commandline/latest/nuget.exe
ARG MY_PRIVATE_NUGET_SOURCE_PWD
COPY NuGet.config ./
RUN c:\nuget.exe source update -ConfigFile .\NuGet.config -Name my-private-nuget-source-name -username my-private-nuget-source-username -StorePasswordInClearText -password $env:MY_PRIVATE_NUGET_SOURCE_PWD
# Builder image
FROM microsoft/dotnet:2.0-sdk AS build
WORKDIR /app
# copy csproj and restore as distinct layers
COPY *.csproj ./
COPY --from=nuget-config NuGet.config ./
RUN dotnet restore
COPY * ./
RUN dotnet publish -c Release -o out
FROM microsoft/dotnet:2.0-runtime AS runtime
WORKDIR /app
COPY --from=publish /app/out ./
ENTRYPOINT ["dotnet", "my-assembly.dll"]Build cmd: docker build -t test --build-arg MY_PROXY_PWD=secret_pwd .
# escape=`
# NuGet config image - NuGet CLI has to be run in a WindowsServerCore container, as a result, PWD must be stored in clear text in order to be used in a later stage. The NuGet config won't be in the runtime image so this shouldn't be an issue.
FROM microsoft/windowsservercore:1709 AS nuget-config
SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]
RUN Invoke-WebRequest -OutFile nuget.exe https://dist.nuget.org/win-x86-commandline/latest/nuget.exe
ARG MY_PROXY_PWD
COPY NuGet.config ./
RUN c:\nuget.exe config -configFile ./nuget.config -set http_proxy=https://proxy.server.com:8080 -set http_proxy.user=my-proxy-user -set x_http_proxy.password=$env:MY_PROXY_PWD; `
# Workaround nuget config limitation for not supporting -StorePasswordInClearText option
(Get-Content NuGet.config) | ForEach-Object { $_ -replace 'x_http_proxy.password', 'http_proxy.password' } | Set-Content NuGet.config
# Builder image
FROM microsoft/dotnet:2.0-sdk AS build
WORKDIR /app
# copy csproj and restore as distinct layers
COPY *.csproj ./
COPY --from=nuget-config NuGet.config ./
RUN dotnet restore
COPY * ./
RUN dotnet publish -c Release -o out
FROM microsoft/dotnet:2.0-runtime AS runtime
WORKDIR /app
COPY --from=publish /app/out ./
ENTRYPOINT ["dotnet", "my-assembly.dll"]
prisar
commented
6 years ago @MichaelSimons Can you provide an way to do the same using docker-compose.
Because I cannot use docker build -t test --build-arg MY_PROXY_PWD=secret_pwd .
AlexanderSysoev
commented
6 years ago @MichaelSimons can you give an example for linux container?
ghost
commented
6 years ago @MichaelSimons can you give an example for linux container?
Nuget cmd will generate a configuration file like this
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<config>
<add key="http_proxy" value="http://my_repo:8080" />
<add key="http_proxy.user" value="my_repo_user" />
<add key="x_http_proxy.password" value="my_repo_password" />
</config>
<packageSources>
<add key="nuget.org" value="https://api.nuget.org/v3/index.json" protocolVersion="3" />
...
</packageSources>
</configuration>So you can populate this file manually and pass to dotnet restore --configfile your_config_file
GersonDias
commented
6 years ago that's terrible... I need to download a 2GB image just to do a nuget sources add.... should be easier... =/
Im5tu
commented
6 years ago @GersonDias I essentially do this:
RUN echo "<configuration><packageSources><add key=\"dotnet-core\" value=\"https://dotnet.myget.org/F/dotnet-core/api/v3/index.json\" /></packageSources></configuration>" > nuget.configAnd make sure the that the credentials etc needed for the source are in that echo statement (passed into the docker build using --build-arg var=val
No need to download the world to do this. Tested as working on an netcore alpine image as part of a multi-stage build.
mdecastilho
commented
5 years ago In Docker for Windows, you can use @GersonDias solution, but you have to escape the "<" characters with "^" (https://stackoverflow.com/a/7308614/189429).
... and you may have problems passing build-args: https://github.com/docker/for-win/issues/542
SychevIgor
commented
5 years ago Azure devops thread about similar question, but in a context of Azure DevOPS (VSTS) https://github.com/Microsoft/azure-pipelines-tasks/issues/6135
Gladskih
commented
5 years ago I'm not sure if I'm stupid or it's just insane complexity (or both).
jholovacs
commented
5 years ago How does anyone expect adoption of Docker for small enterprise apps when it's so difficult to do something really simple?
MichaelSimons
commented
5 years ago We recently published Managing NuGet Credentials in Docker Scenarios. Let us know if it is helpful. If not, tell us about your scenario.
ronnyek
commented
4 years ago With this scenario https://github.com/dotnet/dotnet-docker/blob/master/samples/snippets/nuget-credentials.md#using-the-azure-artifact-credential-provider
Should I need to actually set access token / creds for local testing? Is this something I'd specifically need to change token stuff in the context of a image build in azure devops pipeline?
Seems to me like scenarios like this should be part of visual studio docker tools. Best I've been able to achieve thus far, is a 401 trying to connect to my azure devops hosted feed locally.
I know that article section states I need to set FEED_ACCESSTOKEN, but does this mean I need to go get a PAT and set this manually before I start trying to build locally?
ppatidar-conga
commented
4 years ago you can do following :
1) copy your system's NuGet.Config in project folder at same root level where .csproject is.
2) now in docker file put these statements just before you try to restore package:
COPY ./NuGet.Config ./
3) after that , append the config file location in dotnet restore command like this :
RUN dotnet restore <CS_project_name>.csproj --configfile ./NuGet.Config
4) Now do rest of the thing which you wanted to do .
5) just at the end before entry point or before copying to other container(in case of multistage build) , it is good idea to remove NuGet.Config,as we don’t want that to be available in pod/container to be seen
RUN rm ./NuGet.Config
MichaelSimons
commented
4 years ago @ppatidar-apttus - In regards to the following:
just at the end before entry point or before copying to other container(in case of multistage build) , it is good idea to remove NuGet.Config,as we don’t want that to be available in pod/container to be seen
You must utilize a multi-stage build in this scenario otherwise the NuGet.Config will be in the layers of the resulting image even though you deleted it in a later layer. This is obviously not what you want if your NuGet.config has secrets stored in it.
mldisibio
commented
4 years ago The above solutions, ignoring the authentication issues, assume even the private nuget feed is reachable via an http url. But nuget supports a simple file based private repository as well and this will still fail.
Essentially, if my repository is at C:\dev\nuget-packages this works just fine for all scenarios except issuing a dotnet restore from within the Dockerfile. Even with the Nuget.Config copied, the docker build process sets the repository path as /src/C:\dev\nuget-packages. This use-case seems to leave no other option except as @krakorj mentioned to simply use a pre-published set of source files and not get the advantage of the docker build pipeline...
A few more steps are also possible:
run dotnet restore before running the docker-compose command, and use the --packages option to save the restored packages to the solution folder
e.g. dotnet-restore C:\slnfolder\myproj\myapp.csproj --packages C:\slnfolder\packages
then in the Dockerfile, assuming context is the solution folder and WORKDIR is '/src'
COPY packages/. packages/.
and in the Dockerfile restore line
RUN dotnet restore "myproj/myapp.csproj" -s /src/packages -s https://api.nuget.org/v3/index.json
mirsahibmovsumov
commented
4 years ago @MichaelSimons great work, thanks a lot :)
philipblaquiere
commented
4 years ago Don't know if this still applies, but since .NET Core 3.1.200, Microsoft has added the ability to add nuget sources directly from dotnet CLI via
dotnet nuget add source see https://docs.microsoft.com/en-us/dotnet/core/tools/dotnet-nuget-add-source
tjgalama
commented
3 years ago Thanks to @philipblaquiere I added the following lines to the DockerFile before line RUN dotnet restore:
ARG FEED_SOURCE
ARG FEED_ACCESSTOKEN
RUN dotnet new nugetconfig
RUN dotnet nuget add source %FEED_SOURCE% -n PrivateFeed -u docker -p %FEED_ACCESSTOKEN% --store-password-in-clear-text --configfile nuget.configFor security reason the nuget.config should be removed after running the restore action this should be done within a multi-stage build like @MichaelSimons mentioned here
mthalman
commented
3 years ago For security reason the nuget.config should be removed after running the restore action.
@tjgalama - Please see the comment above that simply deleting the nuget.config file in a separate layer isn't necessarily a secure practice.
danijam
commented
3 years ago With this scenario https://github.com/dotnet/dotnet-docker/blob/master/samples/snippets/nuget-credentials.md#using-the-azure-artifact-credential-provider
Should I need to actually set access token / creds for local testing? Is this something I'd specifically need to change token stuff in the context of a image build in azure devops pipeline?
Seems to me like scenarios like this should be part of visual studio docker tools. Best I've been able to achieve thus far, is a 401 trying to connect to my azure devops hosted feed locally.
I know that article section states I need to set FEED_ACCESSTOKEN, but does this mean I need to go get a PAT and set this manually before I start trying to build locally?
@ronnyek Did you ever get an answer to this? I'm fine with a secret (PAT token being injected during a pipeline. Is there a better experience for local building that doesn't require the user to have to go and make a PAT....
jtrimm007
commented
1 year ago Thanks to @philipblaquiere I added the following lines to the DockerFile before line
RUN dotnet restore:ARG FEED_SOURCE ARG FEED_ACCESSTOKEN RUN dotnet new nugetconfig RUN dotnet nuget add source %FEED_SOURCE% -n PrivateFeed -u docker -p %FEED_ACCESSTOKEN% --store-password-in-clear-text --configfile nuget.configFor security reason ~the nuget.config should be removed after running the restore action~ this should be done within a multi-stage build like @MichaelSimons mentioned here
This worked great for me! Thank you!
stepheninnes
commented
1 year ago Thanks to @philipblaquiere I added the following lines to the DockerFile before line
RUN dotnet restore:ARG FEED_SOURCE ARG FEED_ACCESSTOKEN RUN dotnet new nugetconfig RUN dotnet nuget add source %FEED_SOURCE% -n PrivateFeed -u docker -p %FEED_ACCESSTOKEN% --store-password-in-clear-text --configfile nuget.configFor security reason ~the nuget.config should be removed after running the restore action~ this should be done within a multi-stage build like @MichaelSimons mentioned here
This worked great for me! Thank you!
Thank you so much! I've wasted way too many hours today trying to figure this out.
Dotnet restore command fails from Dockerfile for a private NuGet feed. Note, the private Nuget repository URL is reachable outside docker (i.e. we can restore the project refs using the dotnet restore command localy). Unfortunately the docker dotnet restore fails.
Steps to reproduce the issue
copy csproj and restore as distinct layers
COPY *.csproj ./
copy nuget.config
COPY *.config ./
show files
RUN ls
Restore using local nuget.config or --source attribute
RUN dotnet restore
RUN dotnet restore -s http://s900b216/SEM.NuGetServer.v3/feed/index.json -s https://api.nuget.org/v3/index.json --packages packages --ignore-failed-sources
copy and build everything else
COPY . ./ RUN dotnet publish -c Release -o out ENTRYPOINT ["dotnet", "out/Core.DR-Readout.Create.FromSensor.dll"]
<?xml version="1.0" encoding="utf-8"?>
docker build -t kapsch/sem-iot/microservice/core/dr-readout-create-from-sensor:0.0.1 .
Step 1/10 : FROM microsoft/dotnet:2.0-sdk ---> f29a4c01c987 Step 2/10 : WORKDIR /app ---> Using cache ---> 80437412b8e2 Step 3/10 : COPY .csproj ./ ---> a60776999021 Step 4/10 : COPY .config ./ ---> 263a35fbde94 Step 5/10 : RUN ls ---> Running in e3f9398fbeec Directory: C:\app Mode LastWriteTime Length Name
-a---- 3/21/2018 1:47 PM 1691 Core.DR-Readout.Create.FromSen sor.csproj -a---- 4/4/2018 11:27 AM 424 nuget.config Removing intermediate container e3f9398fbeec ---> 1f7731c6763d Step 6/10 : RUN dotnet restore ---> Running in 4de60bf1fa5e Restoring packages for C:\app\Core.DR-Readout.Create.FromSensor.csproj... C:\Program Files\dotnet\sdk\2.1.101\NuGet.targets(104,5): error : Unable to load the service index for source http://private-repository/SEM.NuGetServer.v3/feed/index.json. [C:\app\Core.DR-Readout.Create.FromSensor.csproj] C:\Program Files\dotnet\sdk\2.1.101\NuGet.targets(104,5): error : An error occurred while sending the request. [C:\app\Core.DR-Readout.Create.FromSensor.csproj] C:\Program Files\dotnet\sdk\2.1.101\NuGet.targets(104,5): error : The server name or address could not be resolved [C:\app\Core.DR-Readout.Create.FromSensor.csproj] ERROR: Service 'core-dr-readout-create-fromsensor' failed to build: The command 'powershell -Command $ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue'; dotnet restore' returned a non-zero code: 1
Step 1/9 : FROM microsoft/dotnet:2.0-sdk ---> f29a4c01c987 Step 2/9 : WORKDIR /app ---> Using cache ---> 80437412b8e2 Step 3/9 : COPY .csproj ./ ---> Using cache ---> a60776999021 Step 4/9 : COPY .config ./ ---> Using cache ---> 263a35fbde94 Step 5/9 : RUN ls ---> Using cache ---> 1f7731c6763d Step 6/9 : RUN dotnet restore -s http://private-repository/SEM.NuGetServer.v3/feed/index.json -s https://api.nuget.org/v3/index.json --packages packages --ignore-failed-sources ---> Running in 156d4c4cf7cb Restoring packages for C:\app\Core.DR-Readout.Create.FromSensor.csproj... C:\Program Files\dotnet\sdk\2.1.101\NuGet.targets(104,5): error : Unable to load the service index for source http://s900b216/SEM.NuGetServer.v3/feed/index.json. [C:\app\Core.DR-Readout.Create.FromSensor.csproj] C:\Program Files\dotnet\sdk\2.1.101\NuGet.targets(104,5): error : An error occurred while sending the request. [C:\app\Core.DR-Readout.Create.FromSensor.csproj] C:\Program Files\dotnet\sdk\2.1.101\NuGet.targets(104,5): error : The server name or address could not be resolved [C:\app\Core.DR-Readout.Create.FromSensor.csproj] ERROR: Service 'core-dr-readout-create-fromsensor' failed to build: The command 'powershell -Command $ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue'; dotnet restore -s http://s900b216/SEM.NuGetServer.v3/feed/index.json -s https://api.nuget.org/v3/index.json --packages packages --ignore-failed-sources' returned a non-zero code: 1
Client: Version: 18.03.0-ce API version: 1.37 Go version: go1.9.4 Git commit: 0520e24 Built: Wed Mar 21 23:06:28 2018 OS/Arch: windows/amd64 Experimental: false Orchestrator: swarm
Server: Engine: Version: 18.03.0-ce API version: 1.37 (minimum version 1.24) Go version: go1.9.4 Git commit: 0520e24 Built: Wed Mar 21 23:21:06 2018 OS/Arch: windows/amd64 Experimental: false
Containers: 10 Running: 0 Paused: 0 Stopped: 10 Images: 17 Server Version: 18.03.0-ce Storage Driver: windowsfilter Windows: Logging Driver: json-file Plugins: Volume: local Network: ics l2bridge l2tunnel nat null overlay transparent Log: awslogs etwlogs fluentd gelf json-file logentries splunk syslog Swarm: inactive Default Isolation: hyperv Kernel Version: 10.0 15063 (15063.0.amd64fre.rs2_release.170317-1834) Operating System: Windows 10 Enterprise OSType: windows Architecture: x86_64 CPUs: 8 Total Memory: 7.446GiB Name: N051B006 ID: 53O4:UEB2:MAK7:STF3:TGSB:3A7E:YRRZ:644W:6PWT:JHK3:ZWVC:CA7C Docker Root Dir: C:\ProgramData\Docker Debug Mode (client): false Debug Mode (server): true File Descriptors: -1 Goroutines: 27 System Time: 2018-04-04T14:13:58.0225934+02:00 EventsListeners: 1 Registry: https://index.docker.io/v1/ Labels: Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false
.NET Command Line Tools (2.1.102)
Product Information: Version: 2.1.102 Commit SHA-1 hash: 8d409357db
Runtime Environment: OS Name: Windows OS Version: 10.0.15063 OS Platform: Windows RID: win10-x64 Base Path: C:\Program Files\dotnet\sdk\2.1.102\
Microsoft .NET Core Shared Framework Host
Version : 2.0.6 Build : 74b1c703813c8910df5b96f304b0f2b78cdf194d
Microsoft Visual Studio Community 2017 Version 15.6.3 VisualStudio.15.Release/15.6.3+27428.2011 Microsoft .NET Framework Version 4.7.02046
Installed Version: Community
Visual C++ 2017 00369-60000-00001-AA962 Microsoft Visual C++ 2017
Visual F# Tools 10.1 for F# 4.1 00369-60000-00001-AA962 Microsoft Visual F# Tools 10.1 for F# 4.1
AnyStatus 1.0 AnyStatus Detailed Info
Application Insights Tools for Visual Studio Package 8.11.10212.1 Application Insights Tools for Visual Studio
ASP.NET and Web Tools 2017 15.0.40314.0 ASP.NET and Web Tools 2017
ASP.NET Core Razor Language Services 1.0 Provides languages services for ASP.NET Core Razor.
ASP.NET Web Frameworks and Tools 2017 5.2.51214.0 For additional information, visit https://www.asp.net/
Azure App Service Tools v3.0.0 15.0.40215.0 Azure App Service Tools v3.0.0
C# Tools 2.7.0-beta3-62707-11. Commit Hash: 75dfc9b33ed624dff3985c7435c902c3c58c0e5c C# components used in the IDE. Depending on your project type and settings, a different version of the compiler may be used.
Common Azure Tools 1.10 Provides common services for use by Azure Mobile Services and Microsoft Azure Tools.
JavaScript Language Service 2.0 JavaScript Language Service
JavaScript Project System 2.0 JavaScript Project System
JavaScript UWP Project System 2.0 JavaScript UWP Project System
Microsoft Azure Tools 2.9 Microsoft Azure Tools for Microsoft Visual Studio 2017 - v2.9.51212.2
Microsoft Continuous Delivery Tools for Visual Studio 0.3 Simplifying the configuration of continuous build integration and continuous build delivery from within the Visual Studio IDE.
Microsoft JVM Debugger 1.0 Provides support for connecting the Visual Studio debugger to JDWP compatible Java Virtual Machines
Microsoft MI-Based Debugger 1.0 Provides support for connecting Visual Studio to MI compatible debuggers
Microsoft Visual C++ Wizards 1.0 Microsoft Visual C++ Wizards
Microsoft Visual Studio Tools for Containers 1.1 Develop, run, validate your ASP.NET Core applications in the target environment. F5 your application directly into a container with debugging, or CTRL + F5 to edit & refresh your app without having to rebuild the container.
Microsoft Visual Studio VC Package 1.0 Microsoft Visual Studio VC Package
NuGet Package Manager 4.6.0 NuGet Package Manager in Visual Studio. For more information about NuGet, visit http://docs.nuget.org/.
ProjectServicesPackage Extension 1.0 ProjectServicesPackage Visual Studio Extension Detailed Info
SQL Server Data Tools 15.1.61801.210 Microsoft SQL Server Data Tools
TypeScript Tools 15.6.20202.3 TypeScript Tools for Microsoft Visual Studio
Visual Basic Tools 2.7.0-beta3-62707-11. Commit Hash: 75dfc9b33ed624dff3985c7435c902c3c58c0e5c Visual Basic components used in the IDE. Depending on your project type and settings, a different version of the compiler may be used.
Visual Studio Code Debug Adapter Host Package 1.0 Interop layer for hosting Visual Studio Code debug adapters in Visual Studio
Visual Studio Tools for Universal Windows Apps 15.0.27428.01 The Visual Studio Tools for Universal Windows apps allow you to build a single universal app experience that can reach every device running Windows 10: phone, tablet, PC, and more. It includes the Microsoft Windows 10 Software Development Kit.