search

IntersectMBO / cardano-node

The core component that is used to participate in a Cardano decentralised blockchain.
https://cardano.org
Apache License 2.0
3.06k stars 720 forks source link

[FR] - Move to support GHC 8.10.1 to support new architectures #1484

Closed mark-stopka closed 4 years ago

mark-stopka commented 4 years ago

Exernal

Describe the feature you'd like Currently cardano-node relies heavily on dependencies that require GHC 8.6.5 , this version is considered obsolete by all major Enterprise Linux providers. This ticket is created to draw a path towards support of GHC 8.10.1 which will be the lowest version of supported GHC for new releases of Enterprise Linux (from all major vendors) versions including support for s390x and riscv64 architectures.

Describe alternatives you've considered Back-porting support for riscv64 to 8.6.5 and getting customer software support agreement with any major Enterprise Linux distribution provider. Cost burden of such endeavour is in in 6 to 7 figures range and thus not feasible for merely a workaround.

Additional context This is a tracking ticket, I would be very grateful if you could create a tracking milestone, it's not an imminent need, so please mark as low priority, but it is certainly something we need to take into consideration moving forward. I have allocated budget for exploration of the work required and would like to use this ticket to link all sub-sequential tickets.

If this is not feasible from your side please let me know and I will create a custom repository and track it there, but I would prefer it to be part of the upstream...

Why is RISC-V 64 support important RISC-V is together with OpenPOWER the only open-source, royalty-free ISA (Instruction Set Architecture) that we can ensure security for the future, also, currently RISC-V Linux capable radiation hardened CPU core is being designed by 2 different chip designers, one of them being SkyWater which also recently open-sourced their PDK for 130nm nodes. All other microchips utilising different proprietary ISAs have major security flaws, either by design1,2, or accidentally3,4,5 as complexity of modern silicon grows.

It is important to add capabilities to support open, royalty-free architecture support to increase resilience against Advanced Persistent Threats (APT) from state-sponsored and state-like actors. In addition, RISC-V already has open-source software defined secure enclave Keystone with hardware secure enclave similar to Intel SGX under development. Utilisation of such capabilities would greatly improve security of Cardano network overall.

Other Resources

Lists of known APT groups

References Management Engines 1: Intel Management Engine 2: AMD Platform Security Processor

Mainstream architecture vulnerabilities 3: Spectre (security vulnerability) 4: Meltdown (security vulnerability) 5: Side Channel Attack Hits Arm CPUs (Update)

mark-stopka commented 4 years ago

@bgamari, you look like the GHC master at Well Typed, so you may have some valuable feedback to add... :)

kevinhammond commented 4 years ago

Hi Mark. We would indeed like to move to a later version of GHC. We have been working to remove this dependency as quickly as we can. I'll let Duncan give you an update on where we are.

mark-stopka commented 4 years ago

You think we could make a list of dependencies that need some work? I can commit a 25% FTE resource to it, he has 5 years of experience in Elixir and two months of training with Haskell, so if you give us some easy fruit to pick-up we will take it...

NOTE: Let's focus on upstream dependencies rather than IOHK developed stuff until the situation clear-up again, as the "issue from the past" got reignited by a recent live-stream between CH and Rick... Hope you guys are not following twitter :)

dcoutts commented 4 years ago

We are well aware of this and we will update at an appropriate point in the development cycle. Obviously that's not just before the Shelley hard fork.

mark-stopka commented 4 years ago

@dcoutts sure, I understand (and support) not doing this at this critical milestone, I just made it to track it and offer a helping hand with the upstream dependencies.

I may also be able to procure s390x LinuxONE VM for testing purposes, not much I can do regarding RISC-V right now, but next year we should be able to provide those also.

kevinhammond commented 4 years ago

Any help is appreciated, of course, Mark!

newhoggy commented 4 years ago

Already did a bit of work here:

mark-stopka commented 4 years ago

New release 1.19.1 still does not build with cabal and GHC 8.10.2, see error below:

cabal: Could not resolve dependencies:
[__0] trying: cardano-node-1.19.1 (user goal)
[__1] trying: brick-0.55 (dependency of cardano-node)
[__2] next goal: base (dependency of cardano-node)
[__2] rejecting: base-4.14.1.0/installed-4.14.1.0 (conflict: brick =>
base<=4.14.0.0)
[__2] rejecting: base-4.14.0.0, base-4.13.0.0, base-4.12.0.0, base-4.11.1.0,
base-4.11.0.0, base-4.10.1.0, base-4.10.0.0, base-4.9.1.0, base-4.9.0.0,
base-4.8.2.0, base-4.8.1.0, base-4.8.0.0, base-4.7.0.2, base-4.7.0.1,
base-4.7.0.0, base-4.6.0.1, base-4.6.0.0, base-4.5.1.0, base-4.5.0.0,
base-4.4.1.0, base-4.4.0.0, base-4.3.1.0, base-4.3.0.0, base-4.2.0.2,
base-4.2.0.1, base-4.2.0.0, base-4.1.0.0, base-4.0.0.0, base-3.0.3.2,
base-3.0.3.1 (constraint from non-upgradeable package requires installed
instance)
[__2] fail (backjumping, conflict set: base, brick, cardano-node)
After searching the rest of the dependency tree exhaustively, these were the
goals I've had most trouble fulfilling: brick, base, cardano-node

@disassembler, maybe update the release notes for 1.19.1?

mrBliss commented 4 years ago

New release 1.19.1 still does not build with cabal and GHC 8.10.2, see error below:

cabal: Could not resolve dependencies:
[__0] trying: cardano-node-1.19.1 (user goal)
[__1] trying: brick-0.55 (dependency of cardano-node)
[__2] next goal: base (dependency of cardano-node)
[__2] rejecting: base-4.14.1.0/installed-4.14.1.0 (conflict: brick =>
base<=4.14.0.0)
[__2] rejecting: base-4.14.0.0, base-4.13.0.0, base-4.12.0.0, base-4.11.1.0,
base-4.11.0.0, base-4.10.1.0, base-4.10.0.0, base-4.9.1.0, base-4.9.0.0,
base-4.8.2.0, base-4.8.1.0, base-4.8.0.0, base-4.7.0.2, base-4.7.0.1,
base-4.7.0.0, base-4.6.0.1, base-4.6.0.0, base-4.5.1.0, base-4.5.0.0,
base-4.4.1.0, base-4.4.0.0, base-4.3.1.0, base-4.3.0.0, base-4.2.0.2,
base-4.2.0.1, base-4.2.0.0, base-4.1.0.0, base-4.0.0.0, base-3.0.3.2,
base-3.0.3.1 (constraint from non-upgradeable package requires installed
instance)
[__2] fail (backjumping, conflict set: base, brick, cardano-node)
After searching the rest of the dependency tree exhaustively, these were the
goals I've had most trouble fulfilling: brick, base, cardano-node

@disassembler, maybe update the release notes for 1.19.1?

Indeed, I think all upstream repos have been updated to work with GHC 8.10.2 (although CI hasn't switched over, so the support could have broken in the meantime), but cardano-node itself hasn't been updated to be compatible with GHC 8.10.2.

mark-stopka commented 4 years ago

@mimi1vx, I think this could be a nice starter if you'd like to join the team as indicated in the email, upstrem dependencies should already have the support...

mimi1vx commented 4 years ago 
[__1] trying: brick-0.55 (dependency of cardano-node)
[__2] next goal: base (dependency of cardano-node)
[__2] rejecting: base-4.14.1.0/installed-4.14.1.0 (conflict: brick =>
base<=4.14.0.0)

brick conflict should befixed in hackage by new revision of brick.cabal file

as stated here https://hackage.haskell.org/package/brick-0.55/revisions/

mimi1vx commented 4 years ago

@mark-stopka @kevinhammond problem here is cabal.project line index-state: 2020-07-15T00:00:00Z

brick revision was pushed to hackage 2020-08-10T18:59:20Z :D

mimi1vx commented 4 years ago

next show stopper is [__1] skipping: base-4.14.0.0, base-4.13.0.0 (has the same characteristics that caused the previous version to fail: excluded by constraint '<4.13' from 'lobemo-backend-ekg')

https://github.com/input-output-hk/iohk-monitoring-framework/blob/master/plugins/backend-ekg/lobemo-backend-ekg.cabal#L22

so we need first: https://github.com/input-output-hk/iohk-monitoring-framework/pull/582

vix-io commented 4 years ago

This is in progress, keep an eye on the release notes

mark-stopka commented 4 years ago

@vix-io, is it solced? Becuase if it isn't it can hardly be closed...

mark-stopka commented 4 years ago

So to answer my own question, no it is not fixed in 1.20.0...

cabal: Could not resolve dependencies:
[__0] trying: lobemo-backend-ekg-0.1.0.1 (user goal)
[__1] next goal: base (dependency of lobemo-backend-ekg)
[__1] rejecting: base-4.14.0.0/installed-4.14.0.0 (conflict:
lobemo-backend-ekg => base<4.13)
[__1] skipping: base-4.14.0.0, base-4.13.0.0 (has the same characteristics
that caused the previous version to fail: excluded by constraint '<4.13' from
'lobemo-backend-ekg')
[__1] rejecting: base-4.12.0.0, base-4.11.1.0, base-4.11.0.0, base-4.10.1.0,
base-4.10.0.0, base-4.9.1.0, base-4.9.0.0, base-4.8.2.0, base-4.8.1.0,
base-4.8.0.0, base-4.7.0.2, base-4.7.0.1, base-4.7.0.0, base-4.6.0.1,
base-4.6.0.0, base-4.5.1.0, base-4.5.0.0, base-4.4.1.0, base-4.4.0.0,
base-4.3.1.0, base-4.3.0.0, base-4.2.0.2, base-4.2.0.1, base-4.2.0.0,
base-4.1.0.0, base-4.0.0.0, base-3.0.3.2, base-3.0.3.1 (constraint from
non-upgradeable package requires installed instance)
[__1] fail (backjumping, conflict set: base, lobemo-backend-ekg)
After searching the rest of the dependency tree exhaustively, these were the
goals I've had most trouble fulfilling: base, lobemo-backend-ekg
mark-stopka commented 4 years ago

Maybe https://github.com/input-output-hk/cardano-node/pull/1874/ will fix it, testing now.

EDIT: Seems there are still some moinor kinks to be ironed out...

/root/.cabal/logs/ghc-8.10.2/cardano-api-1.20.0-8e2d25463f5f4664a11e6ae3adf3dfcd56a0da3b38caa0e1fc212a1aa6077832.log
):
Configuring library for cardano-api-1.20.0..
Preprocessing library for cardano-api-1.20.0..
Building library for cardano-api-1.20.0..
[ 1 of 17] Compiling Cardano.Api.Protocol.Types ( src/Cardano/Api/Protocol/Types.hs, dist/build/Cardano/Api/Protocol/Types.o, dist/build/Cardano/Api/Protocol/Types.dyn_o )
[ 2 of 17] Compiling Cardano.Api.Protocol.Shelley ( src/Cardano/Api/Protocol/Shelley.hs, dist/build/Cardano/Api/Protocol/Shelley.o, dist/build/Cardano/Api/Protocol/Shelley.dyn_o )
[ 3 of 17] Compiling Cardano.Api.Protocol.Cardano ( src/Cardano/Api/Protocol/Cardano.hs, dist/build/Cardano/Api/Protocol/Cardano.o, dist/build/Cardano/Api/Protocol/Cardano.dyn_o )
[ 4 of 17] Compiling Cardano.Api.Protocol.Byron ( src/Cardano/Api/Protocol/Byron.hs, dist/build/Cardano/Api/Protocol/Byron.o, dist/build/Cardano/Api/Protocol/Byron.dyn_o )
[ 5 of 17] Compiling Cardano.Api.Shelley.Genesis ( src/Cardano/Api/Shelley/Genesis.hs, dist/build/Cardano/Api/Shelley/Genesis.o, dist/build/Cardano/Api/Shelley/Genesis.dyn_o )
[ 6 of 17] Compiling Cardano.Api.Shelley.ITN ( src/Cardano/Api/Shelley/ITN.hs, dist/build/Cardano/Api/Shelley/ITN.o, dist/build/Cardano/Api/Shelley/ITN.dyn_o )
[ 7 of 17] Compiling Cardano.Api.TextView ( src/Cardano/Api/TextView.hs, dist/build/Cardano/Api/TextView.o, dist/build/Cardano/Api/TextView.dyn_o )
[ 8 of 17] Compiling Cardano.Api.TxSubmit.ErrorRender ( src/Cardano/Api/TxSubmit/ErrorRender.hs, dist/build/Cardano/Api/TxSubmit/ErrorRender.o, dist/build/Cardano/Api/TxSubmit/ErrorRender.dyn_o )
[ 9 of 17] Compiling Cardano.Api.TxSubmit.Types ( src/Cardano/Api/TxSubmit/Types.hs, dist/build/Cardano/Api/TxSubmit/Types.o, dist/build/Cardano/Api/TxSubmit/Types.dyn_o )
[10 of 17] Compiling Cardano.Api.Typed ( src/Cardano/Api/Typed.hs, dist/build/Cardano/Api/Typed.o, dist/build/Cardano/Api/Typed.dyn_o )
[11 of 17] Compiling Cardano.Api.TxSubmit ( src/Cardano/Api/TxSubmit.hs, dist/build/Cardano/Api/TxSubmit.o, dist/build/Cardano/Api/TxSubmit.dyn_o )

src/Cardano/Api/TxSubmit.hs:29:1: error: [-Wunused-imports, -Werror=unused-imports]
    The import of `StandardShelley'
    from module `Ouroboros.Consensus.Shelley.Protocol.Crypto' is redundant
   |
29 | import           Ouroboros.Consensus.Shelley.Protocol.Crypto (StandardCrypto, StandardShelley)
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
cabal: Failed to build cardano-api-1.20.0 (which is required by
exe:cardano-cli from cardano-cli-1.20.0). See the build log above for details.
mark-stopka commented 4 years ago

@vix-io, please re-open, in addition this issue is about Tier 2 platform support like ARM, s390x and RISC-V, I can't test on these platforms if I can't even get it build on x86_64. @mimi1vx is a Haskell package maintainer at SUSE, if cross-platform support wont't be a major priority for IOG, my company is willing to fund the work, but we do expect support from upstream.

mimi1vx commented 4 years ago

@mark-stopka for libsystemd-journal deps it needs move cabal.project to index-state: 2020-09-25T00:00:00Z ( not needed if is systemd dependency disabled ) and unfortunately -> https://github.com/input-output-hk/cardano-prelude/issues/125

mark-stopka commented 4 years ago

@mark-stopka for libsystemd-journal deps it needs move cabal.project to index-state: 2020-09-25T00:00:00Z ( not needed if is systemd dependency disabled ) and unfortunately -> input-output-hk/cardano-prelude#125

I already got journald-less build via cherry-pick of commit b143dd14c6474d2dba6707057ff465604cbebc6b, I have also got GHC-8.10.2 support via 595ff9f3c0ea77835503ecb67eb854907b338f4f commit.

My next step is to start building for s390x and riscv64 architectures... do you think you could add SLES 12 for SystemZ as a build target to Open Build Service repo, I see you are a maintainer... I would start testing... I have access to following zVM...

systemz1:/home/mark_stopka # cat /proc/cpuinfo
vendor_id       : IBM/S390
# processors    : 2
bogomips per cpu: 3241.00
max thread id   : 0
features        : esan3 zarch stfle msa ldisp eimm dfp edat etf3eh highgprs te vx vxd vxe gs vxe2 vxp sort dflt sie
facilities      : 0 1 2 3 4 6 7 8 9 10 12 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 30 31 32 33 34 35 36 37 38 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 57 58 59 60 61 73 74 75 76 77 80 81 82 128 129 130 131 133 134 135 146 147 148 150 151 152 155 156 168
cache0          : level=1 type=Data scope=Private size=128K line_size=256 associativity=8
cache1          : level=1 type=Instruction scope=Private size=128K line_size=256 associativity=8
cache2          : level=2 type=Data scope=Private size=4096K line_size=256 associativity=8
cache3          : level=2 type=Instruction scope=Private size=4096K line_size=256 associativity=8
cache4          : level=3 type=Unified scope=Shared size=262144K line_size=256 associativity=32
cache5          : level=4 type=Unified scope=Shared size=983040K line_size=256 associativity=60
processor 0: version = FF,  identification = 0F18E8,  machine = 8561
processor 1: version = FF,  identification = 0F18E8,  machine = 8561

cpu number      : 0
cpu MHz dynamic : 5200
cpu MHz static  : 5200

cpu number      : 1
cpu MHz dynamic : 5200
cpu MHz static  : 5200

I currently have no riscv64 HW, but I may use QEMU or OBS for testing... my end goal is to have cardano-node RPMs in OBS repository...

mark-stopka commented 4 years ago

There is a regression in cardano-node 1.21.1 release...

 > [builder 15/16] RUN cabal install cardano-cli       --project-file=cabal.nosystemd.project       --installdir=/usr/local/bin       --install-method=copy       -f -external-libsodium-vrf:               
#22 9.878 cabal: Could not resolve dependencies:                                                                                                                                                            
#22 9.878 [__0] trying: Win32-network-0.1.0.0 (user goal)                                                                                                                                                   
#22 9.879 [__1] trying: base-4.14.1.0/installed-4.14.1.0 (dependency of Win32-network)                                                                                                                      
#22 9.880 [__2] trying: cardano-node-1.21.1 (user goal)                                                                                                                                                     
#22 9.881 [__3] next goal: brick (dependency of cardano-node)
#22 9.882 [__3] rejecting: brick-0.55 (conflict: base==4.14.1.0/installed-4.14.1.0,
#22 9.882 brick => base<=4.14.0.0)
#22 9.884 [__3] skipping: brick-0.54, brick-0.53, brick-0.52.1, brick-0.52, brick-0.51,
#22 9.884 brick-0.50.1, brick-0.50, brick-0.49, brick-0.48, brick-0.47.1, brick-0.47,
#22 9.884 brick-0.46, brick-0.45, brick-0.44.1, brick-0.44, brick-0.43, brick-0.42.1,
#22 9.884 brick-0.42, brick-0.41.5, brick-0.41.4, brick-0.41.3, brick-0.41.2,
#22 9.884 brick-0.41.1, brick-0.41, brick-0.40, brick-0.39, brick-0.38, brick-0.37.2,
#22 9.884 brick-0.37.1, brick-0.37, brick-0.36.3, brick-0.36.2, brick-0.36.1,
#22 9.885 brick-0.36, brick-0.35.1, brick-0.35, brick-0.34.1, brick-0.34, brick-0.33,
#22 9.885 brick-0.32.1, brick-0.32, brick-0.31, brick-0.30, brick-0.29.1, brick-0.29,
#22 9.885 brick-0.28, brick-0.27, brick-0.26.1 (has the same characteristics that caused
#22 9.885 the previous version to fail: excludes 'base' version 4.14.1.0)
#22 9.886 [__3] rejecting: brick-0.26, brick-0.25, brick-0.24.2, brick-0.24.1,
#22 9.886 brick-0.24, brick-0.23, brick-0.22, brick-0.21, brick-0.20.1, brick-0.20,
#22 9.886 brick-0.19, brick-0.18, brick-0.17.2, brick-0.17.1, brick-0.17, brick-0.16,
#22 9.886 brick-0.15.2, brick-0.15.1, brick-0.15, brick-0.14, brick-0.13, brick-0.12,
#22 9.887 brick-0.11, brick-0.10, brick-0.9, brick-0.8, brick-0.7, brick-0.6.4,
#22 9.887 brick-0.6.3, brick-0.6.2, brick-0.6.1, brick-0.6, brick-0.5.1, brick-0.5,
#22 9.887 brick-0.4.1, brick-0.4, brick-0.3.1, brick-0.3, brick-0.2.3, brick-0.2.2,
#22 9.887 brick-0.2.1, brick-0.2, brick-0.1 (constraint from project config TODO
#22 9.887 requires >=0.47)
#22 10.01 [__3] fail (backjumping, conflict set: base, brick, cardano-node)
#22 10.01 After searching the rest of the dependency tree exhaustively, these were the
#22 10.01 goals I've had most trouble fulfilling: brick, base, Win32-network,
#22 10.01 cardano-node
#22 10.01 Try running with --minimize-conflict-set to improve the error message.
#22 10.01 
------
failed to solve with frontend dockerfile.v0: failed to build LLB: executor failed running [/bin/sh -c cabal install cardano-cli       --project-file=cabal.nosystemd.project       --installdir=/usr/local/bin       --install-method=copy       -f -external-libsodium-vrf]: runc did not terminate sucessfully

Build instructions:

FROM perlur/centos-haskell-ghcup AS builder

ENV CABAL_VERSION="3.2.0.0"
ENV GHC_VERSION="8.10.2"

ENV CARDANO_NODE_VERSION="1.21.1"

RUN yum update -y && \
    yum install -y libsodium-devel openssl-devel zlib-devel
RUN ghcup install cabal ${CABAL_VERSION}
RUN ghcup install ghc ${GHC_VERSION} && \
    ghcup set ghc ${GHC_VERSION}

WORKDIR /usr/src
RUN git clone --recurse-submodules https://github.com/input-output-hk/cardano-node
WORKDIR /usr/src/cardano-node
RUN git remote add hasufell https://github.com/hasufell/cardano-node.git
RUN git fetch --all
RUN git checkout ${CARDANO_NODE_VERSION}
COPY cabal.project.local ./
RUN scripts/gen-cabal-nosystemd.sh

# Ugly workaround for vty-5.25.1
RUN ln -s /usr/lib64/libtinfo.so.6 /usr/lib64/libtinfo.so
RUN cabal update
RUN cabal install cardano-cli \
      --project-file=cabal.nosystemd.project \
      --installdir=/usr/local/bin \
      --install-method=copy \
      -f -external-libsodium-vrf
RUN cabal install cardano-node \
      --project-file=cabal.nosystemd.project \
      --installdir=/usr/local/bin \
      --install-method=copy \
      -f -external-libsodium-vrf

FROM perlur/centos-base

LABEL maintainer="Mark Stopka <mark.stopka@perlur.cloud>"

ENV SERVICE_NAME "cardano-node"

RUN yum update -y && \
    yum install -y libsodium jq && \
    yum clean all && \
    dnf clean all && \
    rm -rf /var/cache/yum && \
    rm -rf /var/cache/dnf

EXPOSE 9000

COPY --from=builder /usr/local/bin/cardano-node /usr/local/bin/
COPY --from=builder /usr/local/bin/cardano-cli /usr/local/bin/
COPY usr/local/lib/* /usr/local/lib/
COPY usr/local/bin/* /usr/local/bin/

RUN mkdir -p /srv/cardano/

RUN useradd -c "Cardano node user" \
            -d /srv/cardano/cardano-node/ \
            -m \
            -r \
            -s /bin/nologin \
            cardano-node

RUN mkdir -p /var/run/cardano/
RUN chown cardano-node.cardano-node /var/run/cardano/
USER cardano-node

RUN mkdir /srv/cardano/cardano-node/{etc,log,storage}/
RUN mkdir /srv/cardano/cardano-node/etc/secrets/

WORKDIR /srv/cardano/cardano-node/

COPY srv/cardano/cardano-node/ /srv/cardano/cardano-node/

HEALTHCHECK --interval=1s --timeout=2s CMD cnode-ctl healthcheck || exit 1

ENTRYPOINT ["/usr/local/bin/docker-entrypoint.sh"]

@mimi1vx, @hasufell is that related to https://github.com/input-output-hk/cardano-prelude/issues/125?

hasufell commented 4 years ago

Without having tried the image building yet, this looks like a base constraint conflict. Pass --allow-newer=base to cabal or add that to the cabal.nosystemd.project as allow-newer: base.

Also: I don't think you need this anymore RUN git remote add hasufell https://github.com/hasufell/cardano-node.git. And: cabal.project.local will be ignored, you have to copy it as cabal.nosystemd.project.local into the image.

mark-stopka commented 4 years ago

@hasufell it seems to be working, it is building now, I have also removed the obsolete instructions as advised, thank you a milion!

mark-stopka commented 4 years ago

@hasufell I have got much further with your advice, however I am getting the following error still...

#21 1559. Failed to build typed-protocols-examples-0.1.0.0.
#21 1559. Build log (
#21 1559. /root/.cabal/logs/ghc-8.10.2/typed-protocols-examples-0.1.0.0-2ebc35b9af7e7878b64034f3f61aa7053709c3ca6c867bc8adbfc570341fa6d0.log
#21 1559. ):
#21 1559. Configuring library for typed-protocols-examples-0.1.0.0..
#21 1559. Preprocessing library for typed-protocols-examples-0.1.0.0..
#21 1559. Building library for typed-protocols-examples-0.1.0.0..
#21 1559. [ 1 of 13] Compiling Network.TypedProtocol.Channel ( src/Network/TypedProtocol/Channel.hs, dist/build/Network/TypedProtocol/Channel.o, dist/build/Network/TypedProtocol/Channel.dyn_o )
#21 1559. [ 2 of 13] Compiling Network.TypedProtocol.Codec ( src/Network/TypedProtocol/Codec.hs, dist/build/Network/TypedProtocol/Codec.o, dist/build/Network/TypedProtocol/Codec.dyn_o )
#21 1559. 
#21 1559. src/Network/TypedProtocol/Codec.hs:302:10: error:
#21 1559.     * You can't specify an instance for a tuple constraint
#21 1559.     * In the quantified constraint `forall (st :: ps) (st' :: ps).
#21 1559.                                     (Show (Message ps st st'), Show (ClientHasAgency st),
#21 1559.                                      Show (ServerHasAgency st))'
#21 1559.       In the instance declaration for `Show (AnyMessageAndAgency ps)'
#21 1559.     |
#21 1559. 302 | instance (forall st st'. ( Show (Message ps st st')
#21 1559.     |          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^...
#21 1559. cabal: Failed to build typed-protocols-examples-0.1.0.0 (which is required by
#21 1559. exe:cardano-cli from cardano-cli-1.21.1). See the build log above for details.

My cabal.nosystemd.project.local contains only the following as I am using distro based libsodium for distribution and upstream support (CVEs and such)...

package cardano-crypto-praos
  flags: -external-libsodium-vrf
mark-stopka commented 4 years ago

Same issue on current master...

19 1079. 
#19 1079. Failed to build typed-protocols-examples-0.1.0.0.
#19 1079. Build log (
#19 1079. /root/.cabal/logs/ghc-8.10.2/typed-protocols-examples-0.1.0.0-c04b2a89675a797ae867e81f5f662fb330783f9b15f3754f35cc530fab5d65be.log
#19 1079. ):
#19 1079. Configuring library for typed-protocols-examples-0.1.0.0..
#19 1079. Preprocessing library for typed-protocols-examples-0.1.0.0..
#19 1079. Building library for typed-protocols-examples-0.1.0.0..
#19 1079. [ 1 of 13] Compiling Network.TypedProtocol.Channel ( src/Network/TypedProtocol/Channel.hs, dist/build/Network/TypedProtocol/Channel.o, dist/build/Network/TypedProtocol/Channel.dyn_o )
#19 1079. [ 2 of 13] Compiling Network.TypedProtocol.Codec ( src/Network/TypedProtocol/Codec.hs, dist/build/Network/TypedProtocol/Codec.o, dist/build/Network/TypedProtocol/Codec.dyn_o )
#19 1079. 
#19 1079. src/Network/TypedProtocol/Codec.hs:302:10: error:
#19 1079.     * You can't specify an instance for a tuple constraint
#19 1079.     * In the quantified constraint `forall (st :: ps) (st' :: ps).
#19 1079.                                     (Show (Message ps st st'), Show (ClientHasAgency st),
#19 1079.                                      Show (ServerHasAgency st))'
#19 1079.       In the instance declaration for `Show (AnyMessageAndAgency ps)'
#19 1079.     |
#19 1079. 302 | instance (forall st st'. ( Show (Message ps st st')
#19 1079.     |          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^...
#19 1079. cabal: Failed to build typed-protocols-examples-0.1.0.0 (which is required by
#19 1079. exe:cardano-cli from cardano-cli-1.21.2). See the build log above for details.
#19 1079. 
------
failed to solve with frontend dockerfile.v0: failed to build LLB: executor failed running [/bin/sh -c cabal install cardano-cli       --allow-newer=base       --project-file=cabal.nosystemd.project       --installdir=/usr/local/bin       --install-method=copy       -f -external-libsodium-vrf]: runc did not terminate sucessfully
markstopka@ParentsMacBookAir latest %
mrBliss commented 4 years ago

@mark-stopka That problem was fixed in https://github.com/input-output-hk/ouroboros-network/commit/bcc8a9bab772e2ed1faebdf6a8e1508108164fcc, but it still needs to be propagated to cardano-node, which will be done in https://github.com/input-output-hk/cardano-node/pull/1958.

We haven't officially switched to GHC 8.10 yet. You will at least have to wait until https://github.com/input-output-hk/ouroboros-network/pull/2540 is merged and propagated to cardano-node.

mark-stopka commented 4 years ago

@mrBliss, indeed applying the patch from https://github.com/input-output-hk/ouroboros-network/pull/2652 results in following error...

#20 2353. Failed to build cardano-config-0.1.0.0.
#20 2353. Build log (
#20 2353. /root/.cabal/logs/ghc-8.10.2/cardano-config-0.1.0.0-e882ac7f4db12291b64558a8ed20e0ec05389b350db4712dbb145851b6583324.log
#20 2353. ):
#20 2353. Configuring library for cardano-config-0.1.0.0..
#20 2353. Preprocessing library for cardano-config-0.1.0.0..
#20 2353. Building library for cardano-config-0.1.0.0..
#20 2353. [1 of 2] Compiling Cardano.Config.Git.RevFromGit ( src/Cardano/Config/Git/RevFromGit.hs, dist/build/Cardano/Config/Git/RevFromGit.o, dist/build/Cardano/Config/Git/RevFromGit.dyn_o )
#20 2353. 
#20 2353. src/Cardano/Config/Git/RevFromGit.hs:9:1: error: [-Wunused-imports, -Werror=unused-imports]
#20 2353.     The import of `System.Exit' is redundant
#20 2353.       except perhaps to import instances from `System.Exit'
#20 2353.     To import instances alone, use: import System.Exit()
#20 2353.   |
#20 2353. 9 | import           System.Exit (ExitCode (..))
#20 2353.   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#20 2353. cabal: Failed to build cardano-config-0.1.0.0 (which is required by
#20 2353. exe:cardano-cli from cardano-cli-1.21.2). See the build log above for details.
#20 2353. 
------
failed to solve with frontend dockerfile.v0: failed to build LLB: executor failed running [/bin/sh -c cabal install cardano-cli       --allow-newer=base       --project-file=cabal.nosystemd.project       --installdir=/usr/local/bin       --install-method=copy       -f -external-libsodium-vrf]: runc did not terminate sucessfully
markstopka@ParentsMacBookAir latest %

prior to your rebase, trying now with new commit b6e2e9072be07b5c5e70c06ac0de955d6143783e.

mrBliss commented 4 years ago

@mark-stopka That's just a redundant warning that GHC 8.10 detects, but GHC 8.6 doesn't. You could tell GHC not to turn this warning into an error with -Wno-error=unused-imports.

We are aware of this and will fix all of them (there are tons of them!) when we actually switch over (CI) to GHC 8.10. I think @newhoggy has a draft PR for this somewhere.

mark-stopka commented 4 years ago

@mrBliss I can now confirm that following build works...

FROM perlur/centos-haskell-ghcup AS builder

ENV CABAL_VERSION="3.2.0.0"
ENV GHC_VERSION="8.10.2"

ENV CARDANO_NODE_VERSION="master"

RUN yum update -y && \
    yum install -y libsodium-devel openssl-devel zlib-devel
RUN ghcup install cabal ${CABAL_VERSION}
RUN ghcup install ghc ${GHC_VERSION} && \
    ghcup set ghc ${GHC_VERSION}

WORKDIR /usr/src
RUN git clone --recurse-submodules https://github.com/input-output-hk/cardano-node
WORKDIR /usr/src/cardano-node
RUN git fetch --all
RUN git checkout ${CARDANO_NODE_VERSION}
RUN git cherry-pick -n b6e2e9072be07b5c5e70c06ac0de955d6143783e # Add the upcoming Allegra and Mary eras to CardanoBlock #1958 
RUN scripts/gen-cabal-nosystemd.sh

# Ugly workaround for vty-5.25.1
RUN ln -s /usr/lib64/libtinfo.so.6 /usr/lib64/libtinfo.so
RUN cabal update
RUN cabal install cardano-cli \
      --allow-newer=base \
      --ghc-options="-Wno-error=unused-imports" \ 
      --project-file=cabal.nosystemd.project \
      --installdir=/usr/local/bin \
      --install-method=copy \
      -f -external-libsodium-vrf
RUN cabal install cardano-node \
      --allow-newer=base \
      --ghc-options="-Wno-error=unused-imports" \
      --project-file=cabal.nosystemd.project \
      --installdir=/usr/local/bin \
      --install-method=copy \
      -f -external-libsodium-vrf

FROM perlur/centos-base

LABEL maintainer="Mark Stopka <mark.stopka@perlur.cloud>"

ENV SERVICE_NAME "cardano-node"

RUN yum update -y && \
    yum install -y libsodium jq && \
    yum clean all && \
    dnf clean all && \
    rm -rf /var/cache/yum && \
    rm -rf /var/cache/dnf

EXPOSE 9000

COPY --from=builder /usr/local/bin/cardano-node /usr/local/bin/
COPY --from=builder /usr/local/bin/cardano-cli /usr/local/bin/
COPY usr/local/lib/* /usr/local/lib/
COPY usr/local/bin/* /usr/local/bin/

RUN mkdir -p /srv/cardano/

RUN useradd -c "Cardano node user" \
            -d /srv/cardano/cardano-node/ \
            -m \
            -r \
            -s /bin/nologin \
            cardano-node

RUN mkdir -p /var/run/cardano/
RUN chown cardano-node.cardano-node /var/run/cardano/
USER cardano-node

RUN mkdir /srv/cardano/cardano-node/{etc,log,storage}/
RUN mkdir /srv/cardano/cardano-node/etc/secrets/

WORKDIR /srv/cardano/cardano-node/

COPY srv/cardano/cardano-node/ /srv/cardano/cardano-node/

HEALTHCHECK --interval=1s --timeout=2s CMD cnode-ctl healthcheck || exit 1

ENTRYPOINT ["/usr/local/bin/docker-entrypoint.sh"]

Build log...

$ docker build ./
[+] Building 2568.1s (35/35) FINISHED                                                                                                                                                                       
 => [internal] load build definition from Dockerfile                                                                                                                                                   0.0s
 => => transferring dockerfile: 37B                                                                                                                                                                    0.0s
 => [internal] load .dockerignore                                                                                                                                                                      0.0s
 => => transferring context: 2B                                                                                                                                                                        0.0s
 => [internal] load metadata for docker.io/perlur/centos-base:latest                                                                                                                                   2.3s
 => [internal] load metadata for docker.io/perlur/centos-haskell-ghcup:latest                                                                                                                          1.6s
 => [internal] load build context                                                                                                                                                                      0.0s
 => => transferring context: 550B                                                                                                                                                                      0.0s
 => [builder 1/15] FROM docker.io/perlur/centos-haskell-ghcup@sha256:e4cfae6c397acd826820d9710bab218c6a7410ea6af8f50c817545656747d957                                                                  0.0s
 => [stage-1 1/14] FROM docker.io/perlur/centos-base@sha256:9fdafaf2117b29c677c40aa1b4caf76ef55ccfe901e23fb134f4809ada47f41d                                                                           0.0s
 => CACHED [builder 2/15] RUN yum update -y &&     yum install -y libsodium-devel openssl-devel zlib-devel                                                                                             0.0s
 => CACHED [builder 3/15] RUN ghcup install cabal 3.2.0.0                                                                                                                                              0.0s
 => CACHED [builder 4/15] RUN ghcup install ghc 8.10.2 &&     ghcup set ghc 8.10.2                                                                                                                     0.0s
 => CACHED [builder 5/15] WORKDIR /usr/src                                                                                                                                                             0.0s
 => CACHED [builder 6/15] RUN git clone --recurse-submodules https://github.com/input-output-hk/cardano-node                                                                                           0.0s
 => CACHED [builder 7/15] WORKDIR /usr/src/cardano-node                                                                                                                                                0.0s
 => CACHED [builder 8/15] RUN git fetch --all                                                                                                                                                          0.0s
 => CACHED [builder 9/15] RUN git checkout master                                                                                                                                                      0.0s
 => CACHED [builder 10/15] RUN git cherry-pick -n b6e2e9072be07b5c5e70c06ac0de955d6143783e # Add the upcoming Allegra and Mary eras to CardanoBlock #1958                                              0.0s
 => CACHED [builder 11/15] RUN scripts/gen-cabal-nosystemd.sh                                                                                                                                          0.0s
 => CACHED [builder 12/15] RUN ln -s /usr/lib64/libtinfo.so.6 /usr/lib64/libtinfo.so                                                                                                                   0.0s
 => CACHED [builder 13/15] RUN cabal update                                                                                                                                                            0.0s
 => [builder 14/15] RUN cabal install cardano-cli       --allow-newer=base       --ghc-options="-Wno-error=unused-imports"       --project-file=cabal.nosystemd.project       --installdir=/usr/lo  2501.6s
 => [builder 15/15] RUN cabal install cardano-node       --allow-newer=base       --ghc-options="-Wno-error=unused-imports"       --project-file=cabal.nosystemd.project       --installdir=/usr/loc  59.4s
 => CACHED [stage-1 2/14] RUN yum update -y &&     yum install -y libsodium jq &&     yum clean all &&     dnf clean all &&     rm -rf /var/cache/yum &&     rm -rf /var/cache/dnf                     0.0s 
 => [stage-1 3/14] COPY --from=builder /usr/local/bin/cardano-node /usr/local/bin/                                                                                                                     0.2s 
 => [stage-1 4/14] COPY --from=builder /usr/local/bin/cardano-cli /usr/local/bin/                                                                                                                      0.2s 
 => [stage-1 5/14] COPY usr/local/lib/* /usr/local/lib/                                                                                                                                                0.1s 
 => [stage-1 6/14] COPY usr/local/bin/* /usr/local/bin/                                                                                                                                                0.0s 
 => [stage-1 7/14] RUN mkdir -p /srv/cardano/                                                                                                                                                          0.4s 
 => [stage-1 8/14] RUN useradd -c "Cardano node user"             -d /srv/cardano/cardano-node/             -m             -r             -s /bin/nologin             cardano-node                     0.5s
 => [stage-1 9/14] RUN mkdir -p /var/run/cardano/                                                                                                                                                      0.4s
 => [stage-1 10/14] RUN chown cardano-node.cardano-node /var/run/cardano/                                                                                                                              0.3s
 => [stage-1 11/14] RUN mkdir /srv/cardano/cardano-node/{etc,log,storage}/                                                                                                                             0.3s
 => [stage-1 12/14] RUN mkdir /srv/cardano/cardano-node/etc/secrets/                                                                                                                                   0.3s
 => [stage-1 13/14] WORKDIR /srv/cardano/cardano-node/                                                                                                                                                 0.0s
 => [stage-1 14/14] COPY srv/cardano/cardano-node/ /srv/cardano/cardano-node/                                                                                                                          0.0s
 => exporting to image                                                                                                                                                                                 0.9s
 => => exporting layers                                                                                                                                                                                0.9s
 => => writing image sha256:b7f23dba52de869d3cf8752d18afdbda51b3bca1636b099361603b103afe4a27                                                                                                           0.0s