search

neo-project / neo-modules

MIT License
60 stars 100 forks source link

Aspnet dependencies problems #862

Closed vncoelho closed 8 months ago

vncoelho commented 8 months ago

After https://github.com/neo-project/neo/pull/3065

#35 [31/35] RUN /opt/buildAllList_Plugins_3x.sh
#35 0.342 GOING TO BUILD PLUGIN ApplicationLogs
#35 0.342 
#35 0.342 GETTING PLUGIN NAME AS PARAMETER ApplicationLogs
#35 0.342 GOING TO PUBLISH ApplicationLogs...
#35 0.342 
#35 0.489 MSBuild version 17.7.4+3ebbd7c49 for .NET
#35 0.903   Determining projects to restore...
#35 2.141   Restored /opt/neo-modules/src/RpcServer/RpcServer.csproj (in 122 ms).
#35 2.142   Restored /opt/neo-modules/src/ApplicationLogs/ApplicationLogs.csproj (in 122 ms).
#35 2.143   3 of 5 projects are up-to-date for restore.
#35 2.823   Neo.Json -> /opt/neoLib/src/Neo.Json/bin/Release/net7.0/Neo.Json.dll
#35 2.884   Neo.VM -> /opt/neoLib/src/Neo.VM/bin/Release/net7.0/Neo.VM.dll
#35 2.979   Neo -> /opt/neoLib/src/Neo/bin/Release/net7.0/Neo.dll
#35 3.944 /opt/neo-modules/src/RpcServer/RpcServer.cs(12,17): error CS0234: The type or namespace name 'AspNetCore' does not exist in the namespace 'Microsoft' (are you missing an assembly reference?) [/opt/neo-modules/src/RpcServer/RpcServer.csproj]
#35 3.944 /opt/neo-modules/src/RpcServer/RpcServer.cs(13,17): error CS0234: The type or namespace name 'AspNetCore' does not exist in the namespace 'Microsoft' (are you missing an assembly reference?) [/opt/neo-modules/src/RpcServer/RpcServer.csproj]
#35 3.944 /opt/neo-modules/src/RpcServer/RpcServer.cs(14,17): error CS0234: The type or namespace name 'AspNetCore' does not exist in the namespace 'Microsoft' (are you missing an assembly reference?) [/opt/neo-modules/src/RpcServer/RpcServer.csproj]
#35 3.944 /opt/neo-modules/src/RpcServer/RpcServer.cs(15,17): error CS0234: The type or namespace name 'AspNetCore' does not exist in the namespace 'Microsoft' (are you missing an assembly reference?) [/opt/neo-modules/src/RpcServer/RpcServer.csproj]
#35 3.944 /opt/neo-modules/src/RpcServer/RpcServer.cs(16,17): error CS0234: The type or namespace name 'AspNetCore' does not exist in the namespace 'Microsoft' (are you missing an assembly reference?) [/opt/neo-modules/src/RpcServer/RpcServer.csproj]
#35 3.944 /opt/neo-modules/src/RpcServer/RpcServer.cs(17,28): error CS0234: The type or namespace name 'DependencyInjection' does not exist in the namespace 'Microsoft.Extensions' (are you missing an assembly reference?) [/opt/neo-modules/src/RpcServer/RpcServer.csproj]
#35 3.944 /opt/neo-modules/src/RpcServer/RpcServer.cs(53,32): error CS0246: The type or namespace name 'HttpContext' could not be found (are you missing a using directive or an assembly reference?) [/opt/neo-modules/src/RpcServer/RpcServer.csproj]
#35 3.944 /opt/neo-modules/src/RpcServer/RpcServer.cs(203,40): error CS0246: The type or namespace name 'HttpContext' could not be found (are you missing a using directive or an assembly reference?) [/opt/neo-modules/src/RpcServer/RpcServer.csproj]
#35 3.944 /opt/neo-modules/src/RpcServer/RpcServer.cs(264,57): error CS0246: The type or namespace name 'HttpContext' could not be found (are you missing a using directive or an assembly reference?) [/opt/neo-modules/src/RpcServer/RpcServer.csproj]
#35 3.944 /opt/neo-modules/src/RpcServer/RpcServer.cs(39,17): error CS0246: The type or namespace name 'IWebHost' could not be found (are you missing a using directive or an assembly reference?) [/opt/neo-modules/src/RpcServer/RpcServer.csproj]

There should be other places to fix as well

cschuchardt88 commented 8 months ago

Yes some modules will need updating after this, like RestServer and WebSocketServer

cschuchardt88 commented 8 months ago

See #863, WebSocketServer https://github.com/neo-project/neo-modules/pull/847/commits/43ab998467065a55b8f0edb67fe862dc94d1c673 and RestServer https://github.com/neo-project/neo-modules/pull/839/commits/a072dc12b2dcb5d47753c985f76d38d6a262fad6

vncoelho commented 8 months ago

@cschuchardt88, after the last commit some additional .dlls appearing to be requested.

neo> FATAL [19:29:17.243]   System.IO.FileNotFoundException
Could not load file or assembly 'Microsoft.AspNetCore.Hosting, Version=7.0.0.0,  Culture=neutral,  PublicKeyToken=adb9793829...ch()
   at  System.Thr...eadStart()
Unhandled exception. System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.AspNetCore.Hosting, Version=7.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. The system cannot find the file specified.

File name: 'Microsoft.AspNetCore.Hosting, Version=7.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'
   at Neo.Plugins.RpcServer.StartRpcServer()
   at Neo.Plugins.RpcServerPlugin.OnSystemLoaded(NeoSystem system)
   at Neo.NeoSystem..ctor(ProtocolSettings settings, String storageEngine, String storagePath) in /opt/neoLib/src/Neo/NeoSystem.cs:line 130
   at Neo.CLI.MainService.Start(String[] args) in /opt/neoLib/src/Neo.CLI/CLI/MainService.cs:line 379
   at System.Threading.Tasks.Task.<>c.<ThrowAsync>b__128_1(Object state)
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
/opt/start_node.sh: line 2:   100 Aborted                 (core dumped) dotnet neo-cli.dll
cschuchardt88 commented 8 months ago

Did you see #863? I'll fix this for good. With adding asp.net core to neo-cli. That should fix the problem.

cschuchardt88 commented 8 months ago

As of now you have to publish RpcServer and take all the dlls.

vncoelho commented 8 months ago

As of now you have to publish RpcServer and take all the dlls.

Still produced error

shargon commented 8 months ago

Maybe these dependencies should go in the root directory, not the plugin one

vncoelho commented 8 months ago

That may be possible, @shargon I will try as well.

Lets migrate these modules asap because the problem would had appeared and the PR not merged until fixed.

shargon commented 8 months ago

That may be possible, @shargon I will try as well.

Lets migrate these modules asap because the problem would had appeared and the PR not merged until fixed.

We can revert this PR until solve the dependency problem

vncoelho commented 8 months ago

That may be possible, @shargon I will try as well.

Lets migrate these modules asap because the problem would had appeared and the PR not merged until fixed.

We can revert this PR until solve the dependency problem

It is up to you. I can test here in some minutes. However, even if that works, the requirement would be a quite bad compatible change for generating the plugins.

vncoelho commented 8 months ago 
neo> System.ArgumentException: Value does not fall within the expected range.
   at Neo.SmartContract.Contract.CreateMultiSigRedeemScript(Int32 m, IReadOnlyCollection`1 publicKeys) in /opt/neoLib/src/Neo/SmartContract/Contract.cs:line 106
   at Neo.SmartContract.Contract.GetBFTAddress(IReadOnlyCollection`1 pubkeys) in /opt/neoLib/src/Neo/SmartContract/Contract.cs:line 152
   at Neo.NeoSystem.CreateGenesisBlock(ProtocolSettings settings) in /opt/neoLib/src/Neo/NeoSystem.cs:line 140
   at Neo.NeoSystem..ctor(ProtocolSettings settings, String storageEngine, String storagePath) in /opt/neoLib/src/Neo/NeoSystem.cs:line 122
   at Neo.CLI.MainService.Start(String[] args) in /opt/neoLib/src/Neo.CLI/CLI/MainService.cs:line 379
   at System.Threading.Tasks.Task.<>c.<ThrowAsync>b__128_1(Object state)
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
/opt/start_node.sh: line 2:   110 Aborted                 (core dumped) dotnet neo-cli.dll
root@b7aceb7a4027:/opt/node/Neo.CLI#

@shargon, got this error when dlls were copied to neo-cli folder.

shargon commented 8 months ago 
neo> System.ArgumentException: Value does not fall within the expected range.
   at Neo.SmartContract.Contract.CreateMultiSigRedeemScript(Int32 m, IReadOnlyCollection`1 publicKeys) in /opt/neoLib/src/Neo/SmartContract/Contract.cs:line 106
   at Neo.SmartContract.Contract.GetBFTAddress(IReadOnlyCollection`1 pubkeys) in /opt/neoLib/src/Neo/SmartContract/Contract.cs:line 152
   at Neo.NeoSystem.CreateGenesisBlock(ProtocolSettings settings) in /opt/neoLib/src/Neo/NeoSystem.cs:line 140
   at Neo.NeoSystem..ctor(ProtocolSettings settings, String storageEngine, String storagePath) in /opt/neoLib/src/Neo/NeoSystem.cs:line 122
   at Neo.CLI.MainService.Start(String[] args) in /opt/neoLib/src/Neo.CLI/CLI/MainService.cs:line 379
   at System.Threading.Tasks.Task.<>c.<ThrowAsync>b__128_1(Object state)
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
/opt/start_node.sh: line 2:   110 Aborted                 (core dumped) dotnet neo-cli.dll
root@b7aceb7a4027:/opt/node/Neo.CLI#

@shargon, got this error when dlls were copied to neo-cli folder.

That's seems a different error, maybe related to a mismatch between the config and data stored

vncoelho commented 8 months ago 
neo> System.ArgumentException: Value does not fall within the expected range.
   at Neo.SmartContract.Contract.CreateMultiSigRedeemScript(Int32 m, IReadOnlyCollection`1 publicKeys) in /opt/neoLib/src/Neo/SmartContract/Contract.cs:line 106
   at Neo.SmartContract.Contract.GetBFTAddress(IReadOnlyCollection`1 pubkeys) in /opt/neoLib/src/Neo/SmartContract/Contract.cs:line 152
   at Neo.NeoSystem.CreateGenesisBlock(ProtocolSettings settings) in /opt/neoLib/src/Neo/NeoSystem.cs:line 140
   at Neo.NeoSystem..ctor(ProtocolSettings settings, String storageEngine, String storagePath) in /opt/neoLib/src/Neo/NeoSystem.cs:line 122
   at Neo.CLI.MainService.Start(String[] args) in /opt/neoLib/src/Neo.CLI/CLI/MainService.cs:line 379
   at System.Threading.Tasks.Task.<>c.<ThrowAsync>b__128_1(Object state)
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
/opt/start_node.sh: line 2:   110 Aborted                 (core dumped) dotnet neo-cli.dll
root@b7aceb7a4027:/opt/node/Neo.CLI#

@shargon, got this error when dlls were copied to neo-cli folder.

That's seems a different error, maybe related to a mismatch between the config and data stored

You are correct. In this last tests something got corrupted....Aheuahuea I will double check here.

vncoelho commented 8 months ago

I see...aheuaheuaea I copied the dlls with cp ./Plugins/RpcServer/* ./ Config was overwritten

vncoelho commented 8 months ago

I will do different here and re-run

vncoelho commented 8 months ago 
neo> FATAL [23:21:26.356]   System.IO.FileNotFoundException
Could not load file or assembly 'Microsoft.AspNetCore.Hosting, Version=7.0.0.0,  Culture=neutral,  PublicKeyToken=adb9793829...ch()
   at  System.Thr...eadStart()
Unhandled exception. System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.AspNetCore.Hosting, Version=7.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. The system cannot find the file specified.

File name: 'Microsoft.AspNetCore.Hosting, Version=7.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'
   at Neo.Plugins.RpcServer.StartRpcServer()
   at Neo.Plugins.RpcServerPlugin.OnSystemLoaded(NeoSystem system)
   at Neo.NeoSystem..ctor(ProtocolSettings settings, String storageEngine, String storagePath) in /opt/neoLib/src/Neo/NeoSystem.cs:line 130
   at Neo.CLI.MainService.Start(String[] args) in /opt/neoLib/src/Neo.CLI/CLI/MainService.cs:line 379
   at System.Threading.Tasks.Task.<>c.<ThrowAsync>b__128_1(Object state)
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
/opt/start_node.sh: line 2:   126 Aborted                 (core dumped) dotnet neo-cli.dll

Still the same errors

vncoelho commented 8 months ago

fixed by https://github.com/neo-project/neo/pull/3067#pullrequestreview-1807902350

cschuchardt88 commented 8 months ago

Than your image is corrupt or install

PS: nvm

vncoelho commented 8 months ago

Than your image is corrupt or install

PS: nvm

I did not understand. Everything I use here is very simple, the image is the official. The choice of the image can be wrong, but not broken or corrupted.

cschuchardt88 commented 8 months ago

I was late to the party

shargon commented 8 months ago

@vncoelho could you check if you copy the libraries to root\Plugins it works? if it works we should release the plugin with the libraries there

image

cschuchardt88 commented 8 months ago

This could break self-contained dotnet builds. Buts it works fine with PR #3067. Just need to add to GUI and another exe project

shargon commented 8 months ago

This could break self-contained dotnet builds.

Because they use a different asp net core version? or why?

Buts it works fine with PR #3067. Just need to add to GUI and another exe project

My worries is that we should not only fix the current exe.

cschuchardt88 commented 8 months ago

Well Plugins too have to include framework regardless. Any project that uses the plugins would have to include framework as well.

shargon commented 8 months ago

If tomorrow you want to use netethereum as a dependency (for example), we can't go to neo-node and neo-gui and include this dependency too (neither to neo).

I think that we should fix the plugin release, the problem is if they have a different version of the dependency, but this is easy to solve.

cschuchardt88 commented 8 months ago

Example

Process searching for example.dll on plugin MyPlugin.dll

  1. Path: ./Plugins/MyPlugin/example.dll
  2. Path ./Plugins/example.dll
  3. neo.dll working directory. Example: Path: neo-cli/example.dll
  4. Windows shared library directories. Example: C:\Windows, C:\Windows\system32\, etc
  5. Linux shared library directories. Example: Path: /usr/shared/libs/, etc
  6. MacOS shared library directories.
shargon commented 8 months ago

I think that we don't search in point 1, because the event is static (not related to MyPlugin)

vncoelho commented 8 months ago

I tried to copy to the root path as well, Shargon, but I will try again with the dlls in the Plugins folder.

I do not understand the limitations of the image, @cschuchardt88 . Which one is the recommended and what its official link?

vncoelho commented 8 months ago

@cschuchardt88

https://hub.docker.com/_/microsoft-dotnet-sdk/

image

I am currently using dotnet/aspnet: ASP.NET Core Runtime

I tried with dotnet/runtime: .NET Runtime but it was even more error because it does not have ASP.NET

vncoelho commented 8 months ago

@vncoelho could you check if you copy the libraries to root\Plugins it works? if it works we should release the plugin with the libraries there

image

image

Still error @shargon (without the PR that adds directly to neo-cli)

Unhandled exception. NEO-CLI v3.6.2  -  NEO v3.6.2  -  NEO-VM v3.6.2

neo> System.TypeInitializationException: The type initializer for 'Neo.NeoSystem' threw an exception.
 ---> System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.Extensions.Options, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. The system cannot find the file specified.

File name: 'Microsoft.Extensions.Options, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'
   at System.Reflection.RuntimeAssembly.GetExportedTypes()
   at Neo.Plugins.Plugin.LoadPlugin(Assembly assembly) in /opt/neoLib/src/Neo/Plugins/Plugin.cs:line 158
   at Neo.Plugins.Plugin.LoadPlugins() in /opt/neoLib/src/Neo/Plugins/Plugin.cs:line 190
   at Neo.NeoSystem..cctor() in /opt/neoLib/src/Neo/NeoSystem.cs:line 110
   --- End of inner exception stack trace ---
   at Neo.NeoSystem.CurrentDomain_UnhandledException(Object sender, UnhandledExceptionEventArgs e) in /opt/neoLib/src/Neo/NeoSystem.cs:line 162
/opt/start_node.sh: line 2:    84 Aborted                 (core dumped) dotnet neo-cli.dll
root@ec4eb4fef6b8:/#
shargon commented 8 months ago

Looks like there are some libraries missing there, options are added in the @cschuchardt88 pull request

cschuchardt88 commented 8 months ago

@vncoelho @shargon

Ubuntu Chiseled .NET images are a type of "distroless" container image that contain only the minimal set of packages .NET needs, with everything else removed. These images offer dramatically smaller deployment sizes and attack surface by including only the minimal set of packages required to run .NET applications.

vncoelho commented 8 months ago

@vncoelho @shargon

Ubuntu Chiseled .NET images are a type of "distroless" container image that contain only the minimal set of packages .NET needs, with everything else removed. These images offer dramatically smaller deployment sizes and attack surface by including only the minimal set of packages required to run .NET applications.

"And attack surface"

cschuchardt88 commented 8 months ago

What they are saying there is, The don't load the unnecessary libraries that aren't needed to run. Reason so you don't have bloatware dependencies that are not required to run asp.net core.

vncoelho commented 8 months ago

What they are saying there is, The don't load the unnecessary libraries that aren't needed to run. Reason so you don't have bloatware dependencies that are not required to run asp.net core.

@cschuchardt88, I am kind of naive in this C# frameworks and libraries. I am sorry for that.

I am native from C++ and have developed on that for years. I am having to much problems with this recent change on Aspnet and my enviroment was simple and trustfull for many years here. This change is now requesting some complex changes on some basic client stuff. I believe we need to polish everything better (this can happen in the monorepo). You could push a PR (in the monorepo with modules) with all these changes, being compatible with releases, multiple platforms and etc.

But that is not the reality now. The PR removing Aspnet just crashed the development.

cschuchardt88 commented 8 months ago

Its just my RestServer that is causing this, because of Swagger.

cschuchardt88 commented 8 months ago

@vncoelho Works fine image

whats the problem again?

cschuchardt88 commented 8 months ago

image

cschuchardt88 commented 8 months ago

@shargon @vncoelho

Share with me your dockerfile

Not trying to be mean, but i think your configuration is wrong, I have tried the most basic configuration and it works great

vncoelho commented 8 months ago

@cschuchardt88, It is working for RPCServer after the PR that adds to AspNet. However, still some problems with RestServer I think.

On the other hand, as you should know, this docker has historically been working and adapted for 6 years. It is not perfect, but it is what we used to built clients for us.

# https://hub.docker.com/_/microsoft-dotnet-sdk/
FROM mcr.microsoft.com/dotnet/sdk:7.0.404-jammy
LABEL maintainer="NeoResearch"

# libleveldb-dev libsnappy-dev libc6-dev are used for testing Neo.Plugins.Storage.Tests
RUN apt-get update \
    && apt-get install -y zip libleveldb-dev libsnappy-dev libc6-dev

#==========================================================================
#============== CLONING NEO-CORE ===============================
# get repo, the arguments should be supplied when building using this Dockerfile
ARG NEO_BLOCKCHAIN_URL
ARG LOCAL_NEO_BLOCKCHAIN
RUN mkdir /opt/neoLib 
RUN if [ "$LOCAL_NEO_BLOCKCHAIN" = "false" ] ; then (git clone $NEO_BLOCKCHAIN_URL /opt/neoLib);  fi

ARG NEO_BLOCKCHAIN_BRANCH
ARG NEO_BLOCKCHAIN_COMMIT
RUN if [ "$LOCAL_NEO_BLOCKCHAIN" = "false" ] ; then (cd /opt/neoLib && git pull && git checkout $NEO_BLOCKCHAIN_BRANCH && git checkout $NEO_BLOCKCHAIN_COMMIT);  fi
#==========================================================================
#==========================================================================

#==========================================================================
#============== CHANGING DOCKER FOLDER TO A LOCAL FILE ====================
ARG TEMP_LOCAL_NEO_LIB=/opt/tempLocal_neoLib
RUN mkdir $TEMP_LOCAL_NEO_LIB
ARG LOCAL_NEO_BLOCKCHAIN_URL
ADD $LOCAL_NEO_BLOCKCHAIN_URL $TEMP_LOCAL_NEO_LIB
RUN if [ "$LOCAL_NEO_BLOCKCHAIN" = "true" ] ; then (rm -rf /opt/neoLib/; mv $TEMP_LOCAL_NEO_LIB /opt/neoLib);  fi
#==========================================================================
#==========================================================================

#RUN  dotnet pack /opt/neoLib/src/Neo --configuration Release     --output /opt/neoLib/out
#RUN dotnet pack /opt/neoLib/src/Neo.Json --configuration Release    --output /opt/neoLib/out      
#RUN dotnet pack /opt/neoLib/src/Neo.VM  --configuration Release --output /opt/neoLib/out        
#RUN dotnet pack /opt/neoLib/src/Neo.ConsoleService    --configuration Release      --output /opt/neoLib/out   

#==========================================================================
#========================== ALL THE MAGIC IS HERE =========================
WORKDIR /opt/neoLib/src/Neo.CLI
RUN dotnet restore
RUN dotnet publish -c Release -f net7.0 --output ./
WORKDIR /
#==========================================================================

#==========================================================================
ARG NEO_BLOCKCHAIN_RUN_TESTS
#---------------------- NEO CORE LIB UNIT TESTS ---------------------------
WORKDIR /opt/neoLib
RUN if [ "$NEO_BLOCKCHAIN_RUN_TESTS" = "true" ] ; then (dotnet format --verify-no-changes --verbosity diagnostic);  fi
RUN if [ "$NEO_BLOCKCHAIN_RUN_TESTS" = "true" ] ; then (dotnet test --verbosity n ./tests/Neo.ConsoleService.Tests/Neo.ConsoleService.Tests.csproj);  fi
RUN if [ "$NEO_BLOCKCHAIN_RUN_TESTS" = "true" ] ; then (dotnet test --verbosity n ./tests/Neo.Json.UnitTests/Neo.Json.UnitTests.csproj);  fi
RUN if [ "$NEO_BLOCKCHAIN_RUN_TESTS" = "true" ] ; then (dotnet test --verbosity n ./tests/Neo.UnitTests/Neo.UnitTests.csproj);  fi
RUN if [ "$NEO_BLOCKCHAIN_RUN_TESTS" = "true" ] ; then (dotnet test --verbosity n ./tests/Neo.VM.Tests/Neo.VM.Tests.csproj);  fi
WORKDIR /
#---------------------- NEO CORE LIB UNIT TESTS ---------------------------
#==========================================================================

#==========================================================================
#============= CLONING, PUBLISHING AND ADDING PLUGINS DLL'S ===============
RUN mkdir /opt/neo-modules 

ARG NEO_PLUGINS_URL
ARG LOCAL_NEO_PLUGINS
RUN if [ "$LOCAL_NEO_PLUGINS" = "false" ] ; then (git clone $NEO_PLUGINS_URL /opt/neo-modules);  fi

ARG NEO_PLUGINS_BRANCH
ARG NEO_PLUGINS_COMMIT
RUN if [ "$LOCAL_NEO_PLUGINS" = "false" ] ; then (cd /opt/neo-modules && git pull && git checkout $NEO_PLUGINS_BRANCH && git checkout $NEO_PLUGINS_COMMIT);  fi
#--------------------------------------------------------------------------

#--------------------- CHANGE PLUGINS PATH TO LOCAL FILES -----------------
ARG LOCAL_NEO_PLUGINS_URL
RUN mkdir /opt/tempLocal_neo-modules
ADD $LOCAL_NEO_PLUGINS_URL /opt/tempLocal_neo-modules
RUN if [ "$LOCAL_NEO_PLUGINS" = "true" ] ; then (rm -rf /opt/neo-modules; mv /opt/tempLocal_neo-modules /opt/neo-modules);  fi
#--------------------------------------------------------------------------

#-------- FILES FOR BUILDING & TESTING PUGLINS ----------------------------
ADD neo-modules-publish_scripts/build_plugin_3x.sh /opt/
ADD neo-modules-publish_scripts/test_plugin_3x.sh /opt/
ADD neo-modules-publish_scripts/buildAllList_Plugins_3x.sh /opt/
#--------------------------------------------------------------------------

#-------- PLUGINS TO BE PUBLISHED & TESTED ARE LISTED BELOW ---------------
ARG ENVFILE
ADD $ENVFILE /opt/env-repositories.sh
#--------------------------------------------------------------------------

RUN dotnet remove /opt/neo-modules/Directory.Build.props reference Neo
RUN dotnet add /opt/neo-modules/Directory.Build.props reference /opt/neoLib/src/Neo/Neo.csproj
RUN sed -i 's|\.\.\\neoLib\\src\\Neo\\Neo.csproj|/opt/neoLib/src/Neo/Neo.csproj|' /opt/neo-modules/Directory.Build.props
# STILL NEED TO DO THAT FOR CONSOLE SERVICE ON EACH CSPROJ for EACH PLUGIN

RUN mkdir /opt/neoLib/src/Neo.CLI/Plugins/
RUN /opt/buildAllList_Plugins_3x.sh
#==========================================================================

#==========================================================================
#============ COMPACT NEO-CLI IN A ZIP FILE  ==============================
WORKDIR /opt/neoLib/src/

RUN zip -r /opt/neo-cli-built.zip Neo.CLI
#==========================================================================

# ALL NIGHT LONG - STAY ALIVE
ENTRYPOINT ["tail", "-f", "/dev/null"]
# https://github.com/neo-project/neo/commits/master
NEO_BLOCKCHAIN_URL=https://github.com/neo-project/neo.git
NEO_BLOCKCHAIN_BRANCH=master
NEO_BLOCKCHAIN_COMMIT=87863c6236d60cf3f7e85b90cf7bcbbd898036ae 

# https://github.com/neo-project/neo-modules/commits/master
NEO_PLUGINS_URL=https://github.com/cschuchardt88/neo-modules.git
NEO_PLUGINS_BRANCH=RestServer
NEO_PLUGINS_COMMIT=26170fc

LOCAL_NEO_BLOCKCHAIN="false"
LOCAL_NEO_BLOCKCHAIN_URL="localfiles/neo"
LOCAL_NEO_PLUGINS="false"
LOCAL_NEO_PLUGINS_URL="localfiles/neo-modules"

PLUGINS_LIST_NEO_PACKAGE=(ApplicationLogs LevelDBStore MPTTrie OracleService RocksDBStore RpcClient RestServer RpcServer SQLiteWallet StateService StatesDumper TokensTracker DBFTPlugin)
PLUGINS_UT_LIST=(Neo.Cryptography.MPTTrie.Tests Neo.Network.RPC.Tests Neo.Plugins.OracleService.Tests  Neo.Plugins.RpcServer.Tests  Neo.Plugins.Storage.Tests)

NEO_BLOCKCHAIN_RUN_TESTS="true"
PLUGINS_RUN_TESTS="true"
#!/bin/bash
PLUGIN_TO_INCLUDE=1

function usage {
    echo "Usage: $0 [--plugin-name <plugin to build> --delete-neo-ref <bool>]"
}

while [[ "$#" > 0 ]]; do case $1 in
    -h)
        usage
        exit 0
        ;;
    --plugin-name)
    echo "GETTING PLUGIN NAME AS PARAMETER $2";
        PLUGIN_TO_INCLUDE=$2
        shift; shift
        ;;
    *)
        usage
        exit 1
        ;;
  esac;
done

echo "GOING TO PUBLISH $PLUGIN_TO_INCLUDE..."
echo ""
(cd /opt/neo-modules/src/$PLUGIN_TO_INCLUDE; dotnet restore)
(cd /opt/neo-modules/src/$PLUGIN_TO_INCLUDE; dotnet publish -c Release -f net7.0 -o app)

ls -R /opt/neo-modules/src/$PLUGIN_TO_INCLUDE

ORIGIN_PATH=/opt/neo-modules/src/$PLUGIN_TO_INCLUDE/app

echo "GOING TO CHECK CREATED DLL $PLUGIN_TO_INCLUDE..."
echo ""

if [ ! -f $ORIGIN_PATH/$PLUGIN_TO_INCLUDE.dll ]; then
    echo "PLUGIN DLL File does not exist at $ORIGIN_PATH/$PLUGIN_TO_INCLUDE.dll"
    exit 1
fi

DEST_FOLDER=/opt/neoLib/src/Neo.CLI/Plugins/$PLUGIN_TO_INCLUDE/
mkdir $DEST_FOLDER

echo "GOING TO COPY DEPENDENCIES (IF NEEDED - CURRENTLY JUST ORACLE SERVICES)"
echo ""

# =========================================================
# cp -ri $ORIGIN_PATH/*.dll $DEST_FOLDER
# =========================================================
# OR COPY MANUALLY - TODO FOR SQLIT PLUGIN

if [ $PLUGIN_TO_INCLUDE = "OracleService" ]; then
    echo "Going to copy file $ORIGIN_PATH/$PLUGIN_TO_INCLUDE.dll TO $DEST_FOLDER - and some other dependencies"
    cp -ri $ORIGIN_PATH/$PLUGIN_TO_INCLUDE.dll $DEST_FOLDER
    cp -ri $ORIGIN_PATH/Neo.FileStorage.API.dll $DEST_FOLDER
    cp -ri $ORIGIN_PATH/Grpc.Net.Common.dll $DEST_FOLDER
    cp -ri $ORIGIN_PATH/Grpc.Net.ClientFactory.dll $DEST_FOLDER
    cp -ri $ORIGIN_PATH/Grpc.Net.Client.dll $DEST_FOLDER
    cp -ri $ORIGIN_PATH/Grpc.Core.Api.dll $DEST_FOLDER
    cp -ri $ORIGIN_PATH/Grpc.AspNetCore.Server.dll $DEST_FOLDER
    cp -ri $ORIGIN_PATH/Grpc.AspNetCore.Server.ClientFactory.dll $DEST_FOLDER
    cp -ri $ORIGIN_PATH/Google.Protobuf.dll $DEST_FOLDER
fi

if [ $PLUGIN_TO_INCLUDE = "RestServer" ]; then
    echo "Going to copy file $ORIGIN_PATH/$PLUGIN_TO_INCLUDE.dll TO $DEST_FOLDER - and some other dependencies"
    cp -ri $ORIGIN_PATH/$PLUGIN_TO_INCLUDE.dll $DEST_FOLDER
    cp -ri $ORIGIN_PATH/Microsoft.AspNetCore.JsonPatch.dll $DEST_FOLDER
    cp -ri $ORIGIN_PATH/Microsoft.AspNetCore.Mvc.NewtonsoftJson.dll $DEST_FOLDER
    cp -ri $ORIGIN_PATH/Microsoft.OpenApi.dll $DEST_FOLDER
    cp -ri $ORIGIN_PATH/Swashbuckle.AspNetCore.Swagger.dll $DEST_FOLDER
    cp -ri $ORIGIN_PATH/Swashbuckle.AspNetCore.SwaggerGen.dll $DEST_FOLDER
    cp -ri $ORIGIN_PATH/Swashbuckle.AspNetCore.SwaggerUI.dll $DEST_FOLDER
    cp -ri $ORIGIN_PATH/*.dll $DEST_FOLDER    
    cp -ri $ORIGIN_PATH/RestServer.xml $DEST_FOLDER   
fi

if [ $PLUGIN_TO_INCLUDE = "RpcServer" ]; then
    echo "Going to copy file $ORIGIN_PATH/$PLUGIN_TO_INCLUDE.dll TO $DEST_FOLDER - and some other dependencies"
    cp -ri $ORIGIN_PATH/$PLUGIN_TO_INCLUDE.dll $DEST_FOLDER
    cp -ri $ORIGIN_PATH/*.dll /opt/neoLib/src/Neo.CLI/Plugins
fi

echo "GOING TO COPY DLL ITSELF"
echo ""

if [ $PLUGIN_TO_INCLUDE != "OracleService" ]; then
    echo "Going to copy file /opt/neo-modules/src/$PLUGIN_TO_INCLUDE/app/$PLUGIN_TO_INCLUDE.dll TO /opt/neoNode/neo-cli/Plugins/"
    cp -ri $ORIGIN_PATH/$PLUGIN_TO_INCLUDE.dll $DEST_FOLDER
fi

The generated .zip is usually run with:

# NEO csharp-nodes network simulator
# https://hub.docker.com/_/microsoft-dotnet-aspnet/
FROM mcr.microsoft.com/dotnet/aspnet:7.0.14-jammy

LABEL maintainer="NeoResearch"
LABEL authors="imcoelho,vncoelho"

RUN apt-get update && apt-get install -y --no-install-recommends \
        unzip \
        screen \
        expect \
        libleveldb-dev \
        wget \
        curl \
        nano \
        iputils-ping \
        net-tools \
        iptables \
        psmisc \
        tcpdump \
        iproute2 \
    && rm -rf /var/lib/apt/lists/*

# Add the neo-cli package
ADD ./neo-cli.zip /opt/neo-cli.zip

# Extract and prepare four consensus nodes
RUN unzip -q -d /opt/node /opt/neo-cli.zip

#Script used to start nodes on screen sessions
ADD ./scripts/run.sh /opt/

#Script used to simple start RPC
ADD ./scripts/start_node.sh /opt/

#Script used to modify neo-cli protocol characteristcs - currently, change secondsperblock
#ADD ./scripts/updateConsensusCharacteristics.sh /opt/

# On docker run, start the consensus nodes
CMD ["/bin/bash", "/opt/run.sh"]
cschuchardt88 commented 8 months ago
  1. Copy Source to Paths
    • neo to neo
    • neo-modules to neo-modules
  2. Run in Terminal
    • docker build --no-cache -t neo-docker .
    • docker run --rm -d --network bridge -p 10339:10339 -p 10333:10333 -p 10332:10332 -p 10340:10340 --name test-neo-cli neo-docker
  3. Have fun

Notes:

FROM mcr.microsoft.com/dotnet/sdk:7.0 AS Build

COPY neo /neo
COPY neo-modules /neo-modules

WORKDIR /neo
RUN dotnet restore
RUN dotnet publish src/Neo.CLI --framework net7.0 --no-restore --configuration Release --output /app

COPY libs /app

WORKDIR /neo-modules
RUN dotnet restore
RUN dotnet publish ./src/RestServer --framework net7.0 --no-restore --configuration Release --output /app/Plugins/RestServer
RUN dotnet publish ./src/RpcServer --framework net7.0 --no-restore --configuration Release --output /app/Plugins/RpcServer
RUN dotnet publish ./src/WebSocketServer --framework net7.0 --no-restore --configuration Release --output /app/Plugins/WebSocketServer
RUN dotnet publish ./src/LevelDBStore --framework net7.0 --no-restore --configuration Release --output /app/Plugins/LevelDBStore

FROM mcr.microsoft.com/dotnet/aspnet:7.0 AS Final
RUN apt-get update && apt-get install -y \
  screen \
  libleveldb-dev \
  libsqlite3-dev
RUN rm -rf /var/lib/apt/lists/*

WORKDIR /neo
COPY --from=Build /app .

# RestServer
EXPOSE 10339/tcp
# Node Procotol
EXPOSE 10333/tcp
# RPC Server
EXPOSE 10332/tcp
# WebSocket Server
EXPOSE 10340/tcp

ENTRYPOINT ["screen", "-DmS", "node", "dotnet", "neo-cli.dll", "-r"]
vncoelho commented 8 months ago

@cschuchardt88, I see that the main difference from this docker you send above is:

COPY libs /app, in this case, libs from dotnet/sdk will be in the app folder as well (neo-cli), right? In this necessary?

The --no-restore flag is good for which purpose?

cschuchardt88 commented 8 months ago

you dont need them libs, I install on OS with apt-get. Just comment it out, i forgot to remove.

no-restore is good. I restore in previous lines

vncoelho commented 8 months ago

Ok, @cschuchardt88 This configuration you sent is the same as mine, there are no differences at all in the way we built and run. Just the image that I am using a more specific one aspnet:7.0.14-jammy and the same for sdk.

vncoelho commented 8 months ago

In additional, with this command Publishing direct to neo-cli: 

RUN dotnet publish ./src/RestServer --framework net7.0 --no-restore --configuration Release --output /app/Plugins/RestServer
RUN dotnet publish ./src/RpcServer --framework net7.0 --no-restore --configuration Release --output /app/Plugins/RpcServer
RUN dotnet publish ./src/WebSocketServer --framework net7.0 --no-restore --configuration Release --output /app/Plugins/WebSocketServer
RUN dotnet publish ./src/LevelDBStore --framework net7.0 --no-restore --configuration Release --output /app/Plugins/LevelDBStore

The .zip of the neo-cli will be huge with all .dlls in each Plugin folder. Is this necessary? Would it be better to select the necessary ones?

cschuchardt88 commented 8 months ago

Why are you using zip?

I just tried jammy I changed one line FROM mcr.microsoft.com/dotnet/aspnet:7.0.14-jammy AS Final image

vncoelho commented 8 months ago

We make this full client available and use in other projects as well.

cschuchardt88 commented 8 months ago

HERE IS AN EXAMPLE

Download neo-docker.zip

Directory Structure

|-- Akka.dll
|-- BouncyCastle.Crypto.dll
|-- Data_LevelDB_334F454E
|   |-- 000003.log
|   |-- CURRENT
|   |-- LOCK
|   |-- LOG
|   `-- MANIFEST-000002
|-- K4os.Compression.LZ4.dll
|-- Microsoft.Extensions.Configuration.Abstractions.dll
|-- Microsoft.Extensions.Configuration.Binder.dll
|-- Microsoft.Extensions.Configuration.FileExtensions.dll
|-- Microsoft.Extensions.Configuration.Json.dll
|-- Microsoft.Extensions.Configuration.dll
|-- Microsoft.Extensions.FileProviders.Abstractions.dll
|-- Microsoft.Extensions.FileProviders.Physical.dll
|-- Microsoft.Extensions.FileSystemGlobbing.dll
|-- Microsoft.Extensions.ObjectPool.dll
|-- Microsoft.Extensions.Primitives.dll
|-- Microsoft.Win32.SystemEvents.dll
|-- Neo.ConsoleService.dll
|-- Neo.ConsoleService.pdb
|-- Neo.ConsoleService.xml
|-- Neo.Cryptography.BLS12_381.dll
|-- Neo.Json.dll
|-- Neo.Json.pdb
|-- Neo.Json.xml
|-- Neo.VM.dll
|-- Neo.VM.pdb
|-- Neo.VM.xml
|-- Neo.dll
|-- Neo.pdb
|-- Neo.xml
|-- Newtonsoft.Json.dll
|-- Plugins
|   |-- LevelDBStore
|   |   |-- LevelDBStore.deps.json
|   |   |-- LevelDBStore.dll
|   |   `-- LevelDBStore.pdb
|   |-- RestServer
|   |   |-- Microsoft.AspNetCore.JsonPatch.dll
|   |   |-- Microsoft.AspNetCore.Mvc.NewtonsoftJson.dll
|   |   |-- Microsoft.AspNetCore.Mvc.Versioning.ApiExplorer.dll
|   |   |-- Microsoft.AspNetCore.Mvc.Versioning.dll
|   |   |-- Microsoft.OpenApi.dll
|   |   |-- Neo.ConsoleService.dll
|   |   |-- Newtonsoft.Json.Bson.dll
|   |   |-- Newtonsoft.Json.dll
|   |   |-- RestServer.deps.json
|   |   |-- RestServer.dll
|   |   |-- RestServer.pdb
|   |   |-- RestServer.xml
|   |   |-- Swashbuckle.AspNetCore.Newtonsoft.dll
|   |   |-- Swashbuckle.AspNetCore.Swagger.dll
|   |   |-- Swashbuckle.AspNetCore.SwaggerGen.dll
|   |   |-- Swashbuckle.AspNetCore.SwaggerUI.dll
|   |   |-- System.ServiceProcess.ServiceController.dll
|   |   |-- config.json
|   |   `-- runtimes
|   |       `-- win
|   |           `-- lib
|   |               `-- net6.0
|   |                   `-- System.ServiceProcess.ServiceController.dll
|   |-- RpcServer
|   |   |-- RpcServer.deps.json
|   |   |-- RpcServer.dll
|   |   |-- RpcServer.pdb
|   |   `-- config.json
|   `-- WebSocketServer
|       |-- WebSocketServer.deps.json
|       |-- WebSocketServer.dll
|       |-- WebSocketServer.pdb
|       `-- config.json
|-- System.Configuration.ConfigurationManager.dll
|-- System.Drawing.Common.dll
|-- System.Security.Cryptography.ProtectedData.dll
|-- System.Security.Permissions.dll
|-- System.ServiceProcess.ServiceController.dll
|-- System.Text.Encodings.Web.dll
|-- System.Text.Json.dll
|-- System.Windows.Extensions.dll
|-- config.fs.mainnet.json
|-- config.fs.testnet.json
|-- config.json
|-- config.mainnet.json
|-- config.testnet.json
|-- neo-cli
|-- neo-cli.deps.json
|-- neo-cli.dll
|-- neo-cli.pdb
|-- neo-cli.runtimeconfig.json
|-- neo-cli.xml
`-- runtimes
    |-- browser
    |   `-- lib
    |       `-- net7.0
    |           `-- System.Text.Encodings.Web.dll
    |-- unix
    |   `-- lib
    |       `-- net6.0
    |           `-- System.Drawing.Common.dll
    `-- win
        `-- lib
            |-- net6.0
            |   |-- Microsoft.Win32.SystemEvents.dll
            |   |-- System.Drawing.Common.dll
            |   |-- System.Security.Cryptography.ProtectedData.dll
            |   `-- System.Windows.Extensions.dll
            `-- net7.0
                |-- System.Diagnostics.EventLog.Messages.dll
                `-- System.ServiceProcess.ServiceController.dll

21 directories, 90 files
vncoelho commented 8 months ago

HERE IS AN EXAMPLE

Download neo-docker.zip

Directory Structure

|-- Akka.dll
|-- BouncyCastle.Crypto.dll
|-- Data_LevelDB_334F454E
|   |-- 000003.log
|   |-- CURRENT
|   |-- LOCK
|   |-- LOG
|   `-- MANIFEST-000002
|-- K4os.Compression.LZ4.dll
|-- Microsoft.Extensions.Configuration.Abstractions.dll
|-- Microsoft.Extensions.Configuration.Binder.dll
|-- Microsoft.Extensions.Configuration.FileExtensions.dll
|-- Microsoft.Extensions.Configuration.Json.dll
|-- Microsoft.Extensions.Configuration.dll
|-- Microsoft.Extensions.FileProviders.Abstractions.dll
|-- Microsoft.Extensions.FileProviders.Physical.dll
|-- Microsoft.Extensions.FileSystemGlobbing.dll
|-- Microsoft.Extensions.ObjectPool.dll
|-- Microsoft.Extensions.Primitives.dll
|-- Microsoft.Win32.SystemEvents.dll
|-- Neo.ConsoleService.dll
|-- Neo.ConsoleService.pdb
|-- Neo.ConsoleService.xml
|-- Neo.Cryptography.BLS12_381.dll
|-- Neo.Json.dll
|-- Neo.Json.pdb
|-- Neo.Json.xml
|-- Neo.VM.dll
|-- Neo.VM.pdb
|-- Neo.VM.xml
|-- Neo.dll
|-- Neo.pdb
|-- Neo.xml
|-- Newtonsoft.Json.dll
|-- Plugins
|   |-- LevelDBStore
|   |   |-- LevelDBStore.deps.json
|   |   |-- LevelDBStore.dll
|   |   `-- LevelDBStore.pdb
|   |-- RestServer
|   |   |-- Microsoft.AspNetCore.JsonPatch.dll
|   |   |-- Microsoft.AspNetCore.Mvc.NewtonsoftJson.dll
|   |   |-- Microsoft.AspNetCore.Mvc.Versioning.ApiExplorer.dll
|   |   |-- Microsoft.AspNetCore.Mvc.Versioning.dll
|   |   |-- Microsoft.OpenApi.dll
|   |   |-- Neo.ConsoleService.dll
|   |   |-- Newtonsoft.Json.Bson.dll
|   |   |-- Newtonsoft.Json.dll
|   |   |-- RestServer.deps.json
|   |   |-- RestServer.dll
|   |   |-- RestServer.pdb
|   |   |-- RestServer.xml
|   |   |-- Swashbuckle.AspNetCore.Newtonsoft.dll
|   |   |-- Swashbuckle.AspNetCore.Swagger.dll
|   |   |-- Swashbuckle.AspNetCore.SwaggerGen.dll
|   |   |-- Swashbuckle.AspNetCore.SwaggerUI.dll
|   |   |-- System.ServiceProcess.ServiceController.dll
|   |   |-- config.json
|   |   `-- runtimes
|   |       `-- win
|   |           `-- lib
|   |               `-- net6.0
|   |                   `-- System.ServiceProcess.ServiceController.dll
|   |-- RpcServer
|   |   |-- RpcServer.deps.json
|   |   |-- RpcServer.dll
|   |   |-- RpcServer.pdb
|   |   `-- config.json
|   `-- WebSocketServer
|       |-- WebSocketServer.deps.json
|       |-- WebSocketServer.dll
|       |-- WebSocketServer.pdb
|       `-- config.json
|-- System.Configuration.ConfigurationManager.dll
|-- System.Drawing.Common.dll
|-- System.Security.Cryptography.ProtectedData.dll
|-- System.Security.Permissions.dll
|-- System.ServiceProcess.ServiceController.dll
|-- System.Text.Encodings.Web.dll
|-- System.Text.Json.dll
|-- System.Windows.Extensions.dll
|-- config.fs.mainnet.json
|-- config.fs.testnet.json
|-- config.json
|-- config.mainnet.json
|-- config.testnet.json
|-- neo-cli
|-- neo-cli.deps.json
|-- neo-cli.dll
|-- neo-cli.pdb
|-- neo-cli.runtimeconfig.json
|-- neo-cli.xml
`-- runtimes
    |-- browser
    |   `-- lib
    |       `-- net7.0
    |           `-- System.Text.Encodings.Web.dll
    |-- unix
    |   `-- lib
    |       `-- net6.0
    |           `-- System.Drawing.Common.dll
    `-- win
        `-- lib
            |-- net6.0
            |   |-- Microsoft.Win32.SystemEvents.dll
            |   |-- System.Drawing.Common.dll
            |   |-- System.Security.Cryptography.ProtectedData.dll
            |   `-- System.Windows.Extensions.dll
            `-- net7.0
                |-- System.Diagnostics.EventLog.Messages.dll
                `-- System.ServiceProcess.ServiceController.dll

21 directories, 90 files

I do not see .dlls in this file you sent

cschuchardt88 commented 8 months ago

Run this in terminal in the current directory of the extracted zip file neo-docker.zip.

docker build --no-cache -t neo-docker .
docker run --rm -d --network bridge -p 10339:10339 -p 10333:10333 -p 10332:10332 -p 10340:10340 --name test-neo-cli neo-docker