Run an error in the K8s container

[geffzhang]

My k8s docker environment is as follows: Use up to 1 cores of cpu,1g memory，The following error occurred in the run：

2019-06-02T12:31:51.410400083Z display_data: [{"source":"","data":{"text/plain":"Unable to restore packages from '/tmp/scripts/root/.dotnet/tools/.store/icsharpcore/0.2.2/icsharpcore/0.2.2/tools/netcoreapp2.2/any/REPL/interactive/interactive/script.csproj'. Make sure that all script files contains valid NuGet references\n at Dotnet.Script.DependencyModel.Context.DotnetRestorer.Restore(String pathToProjectFile, String[] packageSources)\n at Dotnet.Script.DependencyModel.Context.CachedRestorer.gRestoreAndCacheProjectFile|5_0(<>c__DisplayClass5_0& )\n at Dotnet.Script.DependencyModel.Context.CachedRestorer.Restore(String pathToProjectFile, String[] packageSources)\n at Dotnet.Script.DependencyModel.Context.ProfiledRestorer.Restore(String pathToProjectFile, String[] packageSources)\n at Dotnet.Script.DependencyModel.Runtime.RuntimeDependencyResolver.GetDependenciesForCode(String targetDirectory, ScriptMode scriptMode, String[] packageSources, String code)\n at ICSharpCore.Script.InteractiveScriptEngine.TryLoadReferenceFromScript(String statement) in /Users/kerryjiang/WorkShop/ICSharpCore/src/ICSharpCore/Script/InteractiveScriptEngine.cs:line 172\n at ICSharpCore.Script.InteractiveScriptEngine.ExecuteAsync(String statement) in /Users/kerryjiang/WorkShop/ICSharpCore/src/ICSharpCore/Script/InteractiveScriptEngine.cs:line 108\n at ICSharpCore.RequestHandlers.ExecuteHandler1.Process(Message1 message) in /Users/kerryjiang/WorkShop/ICSharpCore/src/ICSharpCore/RequestHandlers/ExecuteHandler.cs:line 49","text/html":"<p style=\"color:red;\">Unable to restore packages from '/tmp/scripts/root/.dotnet/tools/.store/icsharpcore/0.2.2/icsharpcore/0.2.2/tools/netcoreapp2.2/any/REPL/interactive/interactive/script.csproj'. Make sure that all script files contains valid NuGet references\n at Dotnet.Script.DependencyModel.Context.DotnetRestorer.Restore(String pathToProjectFile, String[] packageSources)\n at Dotnet.Script.DependencyModel.Context.CachedRestorer.gRestoreAndCacheProjectFile|5_0(<>cDisplayClass5_0& )\n at Dotnet.Script.DependencyModel.Context.CachedRestorer.Restore(String pathToProjectFile, String[] packageSources)\n at Dotnet.Script.DependencyModel.Context.ProfiledRestorer.Restore(String pathToProjectFile, String[] packageSources)\n at Dotnet.Script.DependencyModel.Runtime.RuntimeDependencyResolver.GetDependenciesForCode(String targetDirectory, ScriptMode scriptMode, String[] packageSources, String code)\n at ICSharpCore.Script.InteractiveScriptEngine.TryLoadReferenceFromScript(String statement) in /Users/kerryjiang/WorkShop/ICSharpCore/src/ICSharpCore/Script/InteractiveScriptEngine.cs:line 172\n at ICSharpCore.Script.InteractiveScriptEngine.ExecuteAsync(String statement) in /Users/kerryjiang/WorkShop/ICSharpCore/src/ICSharpCore/Script/InteractiveScriptEngine.cs:line 108\n at ICSharpCore.RequestHandlers.ExecuteHandler1.Process(Message1 message) in /Users/kerryjiang/WorkShop/ICSharpCore/src/ICSharpCore/RequestHandlers/ExecuteHandler.cs:line 49

"},"metadata":{},"transient":{}}] 2019-06-02T12:31:51.410444224Z execute_request: [NetMQMessage[e1001bd436af4fdc8d1a85ba781f11c4,<IDS|MSG>,ea21fe6e2b26602df5b3fb4ba3a1e4ed6399853e067b13b62fa24aa28c9a3a88,{"msg_type":"execute_request","msg_id":"4072fa3832cd493d8ca1589fdee3137e","version":"5.2","username":"username","session":"e1001bd436af4fdc8d1a85ba781f11c4"},{},{},{"user_expressions":{},"code":"var str = \"Hello, TensorFlow.NET!\";\nvar hello = tf.constant(str);\n\n// Start tf session\nusing(var sess = tf.Session())\n{\n // Run the op\n return sess.run(hello);\n}","allow_stdin":true,"store_history":true,"stop_on_error":true,"silent":false}]] 2019-06-02T12:31:51.410452581Z status: [{"execution_state":"busy"}] 2019-06-02T12:31:51.410603242Z fail: ExecuteHandler[0] 2019-06-02T12:31:51.410615665Z Failed to run the code: using Tensorflow; 2019-06-02T12:31:51.410619855Z System.Exception: Unable to restore packages from '/tmp/scripts/root/.dotnet/tools/.store/icsharpcore/0.2.2/icsharpcore/0.2.2/tools/netcoreapp2.2/any/REPL/interactive/interactive/script.csproj'. Make sure that all script files contains valid NuGet references 2019-06-02T12:31:51.41062427Z at Dotnet.Script.DependencyModel.Context.DotnetRestorer.Restore(String pathToProjectFile, String[] packageSources) 2019-06-02T12:31:51.410628387Z at Dotnet.Script.DependencyModel.Context.CachedRestorer.g__RestoreAndCacheProjectFile|5_0(<>cDisplayClass5_0& ) 2019-06-02T12:31:51.410632651Z at Dotnet.Script.DependencyModel.Context.CachedRestorer.Restore(String pathToProjectFile, String[] packageSources) 2019-06-02T12:31:51.410636674Z at Dotnet.Script.DependencyModel.Context.ProfiledRestorer.Restore(String pathToProjectFile, String[] packageSources) 2019-06-02T12:31:51.410640624Z at Dotnet.Script.DependencyModel.Runtime.RuntimeDependencyResolver.GetDependenciesForCode(String targetDirectory, ScriptMode scriptMode, String[] packageSources, String code) 2019-06-02T12:31:51.410644679Z at ICSharpCore.Script.InteractiveScriptEngine.TryLoadReferenceFromScript(String statement) in /Users/kerryjiang/WorkShop/ICSharpCore/src/ICSharpCore/Script/InteractiveScriptEngine.cs:line 172 2019-06-02T12:31:51.410648802Z at ICSharpCore.Script.InteractiveScriptEngine.ExecuteAsync(String statement) in /Users/kerryjiang/WorkShop/ICSharpCore/src/ICSharpCore/Script/InteractiveScriptEngine.cs:line 108 2019-06-02T12:31:51.410661877Z at ICSharpCore.RequestHandlers.ExecuteHandler1.Process(Message1 message) in /Users/kerryjiang/WorkShop/ICSharpCore/src/ICSharpCore/RequestHandlers/ExecuteHandler.cs:line 49

[kerryjiang]

It seems like a nuget package downloading failure. Does your k8s environment allow outgoing traffic?

[geffzhang]

From Microsoft/dotnet, this should be change. https://devblogs.microsoft.com/dotnet/net-core-container-images-now-published-to-microsoft-container-registry/

[geffzhang]

新版本的镜像运行正常了

[geffzhang]

Unable to restore packages from '/tmp/scripts/root/.dotnet/tools/.store/icsharpcore/0.2.2/icsharpcore/0.2.2/tools/netcoreapp2.2/any/REPL/interactive/interactive/script.csproj'. Make sure that all script files contains valid NuGet references at Dotnet.Script.DependencyModel.Context.DotnetRestorer.Restore(String pathToProjectFile, String[] packageSources) at Dotnet.Script.DependencyModel.Context.CachedRestorer.g__RestoreAndCacheProjectFile|5_0(<>c__DisplayClass5_0& ) at Dotnet.Script.DependencyModel.Context.CachedRestorer.Restore(String pathToProjectFile, String[] packageSources) at Dotnet.Script.DependencyModel.Context.ProfiledRestorer.Restore(String pathToProjectFile, String[] packageSources) at Dotnet.Script.DependencyModel.Runtime.RuntimeDependencyResolver.GetDependenciesForCode(String targetDirectory, ScriptMode scriptMode, String[] packageSources, String code) at ICSharpCore.Script.InteractiveScriptEngine.TryLoadReferenceFromScript(String statement) in /Users/kerryjiang/WorkShop/ICSharpCore/src/ICSharpCore/Script/InteractiveScriptEngine.cs:line 172 at ICSharpCore.Script.InteractiveScriptEngine.ExecuteAsync(String statement) in /Users/kerryjiang/WorkShop/ICSharpCore/src/ICSharpCore/Script/InteractiveScriptEngine.cs:line 108 at ICSharpCore.RequestHandlers.ExecuteHandler1.Process(Message1 message) in /Users/kerryjiang/WorkShop/ICSharpCore/src/ICSharpCore/RequestHandlers/ExecuteHandler.cs:line 49

[kerryjiang]

Still the problem about restore packages.

[geffzhang]

It's okay to manually go inside the docker and run the command line.

/tmp/scripts/root/.dotnet/tools/.store/icsharpcore/0.2.2/icsharpcore/0.2.2/tools/netcoreapp2.2/any/REPL/interactive/interactive# ls NuGet.Config obj script.csproj /tmp/scripts/root/.dotnet/tools/.store/icsharpcore/0.2.2/icsharpcore/0.2.2/tools/netcoreapp2.2/any/REPL/interactive/interactive# dotnet restore Restore completed in 622.5 ms for /tmp/scripts/root/.dotnet/tools/.store/icsharpcore/0.2.2/icsharpcore/0.2.2/tools/netcoreapp2.2/any/REPL/interactive/interactive/script.csproj.

[whatevergeek]

@geffzhang I tried running via commandline inside the container.... "using Tensorflow" completed however, whenever i start using tensorflow (e.g. tf.constant("test");), the kernel just dies... any workaround for this?

[kerryjiang]

@whatevergeek which docker image version do you use?

Did you get same error with Geff Zhang?

[whatevergeek]

@whatevergeek which docker image version do you use?

Did you get same error with Geff Zhang? I used scisharpstack/scisharpcube:0.4

Initially the container fails when loading (using Tensorflow;) After i've done @geffzhang commandline restore, it got completed. However, the container dies again when i do: tf.constant("test");

[kerryjiang]

That means the native Tensorflow lib cannot be found. I will try to reproduce it.

[kerryjiang]

Which kind host do you use? Windows, Linux or Mac OSX?

[whatevergeek]

Which kind host do you use? Windows, Linux or Mac OSX?

Windows. Using Docker Toolbox for hosting the image.

[whatevergeek]

Btw, i'm also trying to use binder to help reproduce it online. However, the environment doesn't seem to launch. Anyway, placing my repo here in case anyone got suggestion on the dockerfile setup: https://github.com/whatevergeek/SciSharpCubeBinder

[javiercp]

@whatevergeek Using the Try.NET repository as a starting base, I have made a working dockerfile for Binder:

https://github.com/javiercp/BinderTF.NET

I haven't tested it fully but it seems to work. The helloworld sample returns null but otherwise the objects seem right, and the linear regression test works just fine.

@kerryjiang I have the same problem with the dockerfile from this repo, but I'm running it from behind my corporate firewall. This maybe a problem to get the Nuget packages. I'll try it from home later.

[kerryjiang]

Do you use container in Kubernetes or independent docker container?

[javiercp]

I'm using an independent docker container.. Also on Windows 10.

[kerryjiang]

@javiercp under Windows Linux Subsystem?

[javiercp]

No, just standard Windows 10, but unfortunately with corporate proxy/antivirus/...

At home I'll have a simpler connection. I'll check it when I get back.

[whatevergeek]

@javiercp i tried the binderhub... it kinda works... thanks for sharing... i tried the following:

var str = "Hello, TensorFlow.NET!"; var hello = tf.constant(str);

` var str = "Hello, TensorFlow.NET!"; var hello = tf.constant(str);

// Start tf session using(var sess = tf.Session()) { var result = sess.run(hello); Console.WriteLine(result.GetType()); Console.WriteLine(result.ToString()); // Run the op return result; }`

-- result --

` NumSharp.NDArray

Hello, TensorFlow.NET!

\<null>

`

[javiercp]

Yup, that's what I see too. The return result shows null, although the result object seems fine.

If you try it with a int constant it works without problem: var foo = tf.constant(1); var bar = tf.constant(2);

// Start tf session using(var sess = tf.Session()) { // Run the op var result = sess.run(foo + bar);

Console.WriteLine(result.GetType());
Console.WriteLine(result.ToString());    

return result;

}

-- result -- NumSharp.NDArray 3

index	value
0	3

So I guess it works, it's just something when it tries to show the result.

[javiercp]

@kerryjiang A bit more info. This is the log from the docker container:

[...] 6bcea139de54: Pull complete Digest: sha256:6b518f79038b4eaf385f32a2ad823764ea94d116ac94f4d4c93b68a44366d44f Status: Downloaded newer image for scisharpstack/scisharpcube:latest [I 09:58:37.832 NotebookApp] Writing notebook server cookie secret to /root/.local/share/jupyter/runtime/notebook_cookie_secret [I 09:58:38.018 NotebookApp] Serving notebooks from local directory: /scisharp/home [I 09:58:38.018 NotebookApp] The Jupyter Notebook is running at: [I 09:58:38.018 NotebookApp] http://87e79186a562:8888/?token=c1a49993cbbf76792a7a2bc52658f6365bb7edabcefb6250 [I 09:58:38.018 NotebookApp] or http://127.0.0.1:8888/?token=c1a49993cbbf76792a7a2bc52658f6365bb7edabcefb6250 [I 09:58:38.018 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation). [C 09:58:38.023 NotebookApp]

To access the notebook, open this file in a browser:
    file:///root/.local/share/jupyter/runtime/nbserver-1-open.html
Or copy and paste one of these URLs:
    http://87e79186a562:8888/?token=c1a49993cbbf76792a7a2bc52658f6365bb7edabcefb6250
 or http://127.0.0.1:8888/?token=c1a49993cbbf76792a7a2bc52658f6365bb7edabcefb6250

[I 12:34:56.180 NotebookApp] 302 GET /?token=c1a49993cbbf76792a7a2bc52658f6365bb7edabcefb6250 (172.17.0.1) 0.82ms [I 12:35:00.604 NotebookApp] Writing notebook-signing key to /root/.local/share/jupyter/notebook_secret [W 12:35:00.604 NotebookApp] Notebook samples/HelloWorld.ipynb is not trusted [I 12:35:00.723 NotebookApp] Kernel started: 2c31cc70-8aae-4b39-8790-cc33c3eac300 [W 12:35:00.759 NotebookApp] 404 GET /static/components/codemirror/mode/.netstandard/.netstandard.js?v=20191115095837 (172.17.0.1) 31.31ms referer=http://127.0.0.1:8888/notebooks/samples/HelloWorld.ipynb Kernel connecting... arg 0: /root/.local/share/jupyter/runtime/kernel-2c31cc70-8aae-4b39-8790-cc33c3eac300.json arg 1: /scisharp/refs.txt {"shell_port":33909,"iopub_port":59745,"stdin_port":54997,"control_port":35107,"hb_port":45979,"IP":"127.0.0.1","Key":"ef8cdb3b-c5d50a30318711a08f4c3346","Transport":"tcp","signature_scheme":"hmac-sha256","kernel_name":"csharpcore"} Listening Shell @tcp://127.0.0.1:33909 Listening IOPub @tcp://127.0.0.1:59745 kernel_info_request: [NetMQMessage[ ?~qO,<IDS|MSG>,6d6688c88562e04933708634a15aa3a73587155c1e6ee13fbda8ebc138ed2fcc,{"date":"2019-11-15T12:35:00.764013Z","username":"username","msg_type":"kernel_info_request","session":"bf825497633848628df4be9a602a2a2f","version":"5.3","msg_id":"d71ebcf5-78b257e2cdcf146180cb8a55"},{},{},{}]] kernel_info_request: [NetMQMessage[bf825497633848628df4be9a602a2a2f,<IDS|MSG>,07c43a7d1ddae3a9445a289345df1844c22000d4c32bdfc82d38d80f6a3e3255,{"username":"username","msg_type":"kernel_info_request","msg_id":"5a5b6ae49d9546ef88aebd906f906059","session":"bf825497633848628df4be9a602a2a2f","version":"5.2"},{},{},{}]] comm_info_request: [NetMQMessage[bf825497633848628df4be9a602a2a2f,<IDS|MSG>,147639e491b474a863f349364c76351544d1b3d3ccb63735c220a0ecd53c6d13,{"username":"username","msg_type":"comm_info_request","msg_id":"50c9484b17664c338208be8c415f3992","session":"bf825497633848628df4be9a602a2a2f","version":"5.2"},{},{},{"target_name":"jupyter.widget"}]] execute_request: [NetMQMessage[bf825497633848628df4be9a602a2a2f,<IDS|MSG>,3a5e7482f53a5dde679a5b74a9088dc0a3206a8694fa18c9d059b8c158abb00c,{"username":"username","msg_type":"execute_request","msg_id":"9213303500f14e13af545a2337127bbb","session":"bf825497633848628df4be9a602a2a2f","version":"5.2"},{},{},{"code":"using Tensorflow;","user_expressions":{},"store_history":true,"stop_on_error":true,"allow_stdin":true,"silent":false}]] fail: ExecuteHandler[0] Failed to run the code: using Tensorflow; System.Exception: Unable to restore packages from '/tmp/scripts/root/.dotnet/tools/.store/icsharpcore/0.2.4/icsharpcore/0.2.4/tools/netcoreapp2.2/any/REPL/interactive/interactive/script.csproj'. Make sure that all script files contains valid NuGet references at Dotnet.Script.DependencyModel.Context.DotnetRestorer.Restore(String pathToProjectFile, String[] packageSources) at Dotnet.Script.DependencyModel.Context.CachedRestorer.g__RestoreAndCacheProjectFile|5_0(<>c__DisplayClass5_0& ) at Dotnet.Script.DependencyModel.Context.CachedRestorer.Restore(String pathToProjectFile, String[] packageSources) at Dotnet.Script.DependencyModel.Context.ProfiledRestorer.Restore(String pathToProjectFile, String[] packageSources) at Dotnet.Script.DependencyModel.Runtime.RuntimeDependencyResolver.GetDependenciesForCode(String targetDirectory, ScriptMode scriptMode, String[] packageSources, String code) at ICSharpCore.Script.InteractiveScriptEngine.TryLoadReferenceFromScript(String statement) in /Users/kerryjiang/WorkShop/ICSharpCore/src/ICSharpCore/Script/InteractiveScriptEngine.cs:line 170 at ICSharpCore.Script.InteractiveScriptEngine.ExecuteAsync(String statement) in /Users/kerryjiang/WorkShop/ICSharpCore/src/ICSharpCore/Script/InteractiveScriptEngine.cs:line 106 at ICSharpCore.RequestHandlers.ExecuteHandler1.Process(Message1 message) in /Users/kerryjiang/WorkShop/ICSharpCore/src/ICSharpCore/RequestHandlers/ExecuteHandler.cs:line 70

And this is what I see in the browser:

It seems the restore isn't working, that's why suppose there is a problem with my container reaching nuget.

[Oceania2018]

@javiercp I can run it:

@kerryjiang Can you help update the Cube environment? Seems the syntax and package are out of date.

[Oceania2018]

@javiercp Do you mind if we put your link on our SciSharpCube Readme? I thought it's so convenient to experience SciSharp stack for new developers.

[javiercp]

@Oceania2018 Sure, no problem. I'll also update the repo readme to point that it's using SciSharp's Tensorflow.NET.