search

hyperledger-cacti / cacti

Hyperledger Cacti is a new approach to the blockchain interoperability problem
https://wiki.hyperledger.org/display/cactus
Apache License 2.0
344 stars 286 forks source link

feat(daml): creation of connector class #3615

Open raynatopedrajeta opened 2 weeks ago

raynatopedrajeta commented 2 weeks ago

Commit to be reviewed

feat(daml): creation of connector class

Primary Changes
---------------
1. Create a DAML connector class
2. Created OpenAPI endpoints of DAML
3. Created DAML web services for create, exercise and query contracts
4. Create simple IOU Transaction using DAML

Fixes #3489 


Co-authored-by: Peter Somogyvari <peter.somogyvari@accenture.com>

**Pull Request Requirements**
- [ ] Rebased onto `upstream/main` branch and squashed into single commit to help maintainers review it more efficient and to avoid spaghetti git commit graphs that obfuscate which commit did exactly what change, when and, why.
- [ ] Have git sign off at the end of commit message to avoid being marked red. You can add `-s` flag when using `git commit` command. You may refer to this [link](https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-commits) for more information.
- [ ] Follow the Commit Linting specification. You may refer to this [link](https://www.conventionalcommits.org/en/v1.0.0-beta.4/#specification) for more information. 

**Character Limit**
- [ ] Pull Request Title and Commit Subject must not exceed 72 characters (including spaces and special characters).
- [ ] Commit Message per line must not exceed 80 characters (including spaces and special characters).

**A Must Read for Beginners**
For rebasing and squashing, here's a [must read guide](https://github.com/servo/servo/wiki/Beginner's-guide-to-rebasing-and-squashing) for beginners.
raynatopedrajeta commented 1 week ago

Hello @petermetz ,

May I request for you to update the docker image of this task?: imageVersion: "2024-09-08T07-40-07-dev-2cc217b7a", imageName: "ghcr.io/hyperledger/cacti-daml-all-in-one",

image

Cheers!

Rayn

petermetz commented 1 week ago

Hello @petermetz ,

May I request for you to update the docker image of this task?: imageVersion: "2024-09-08T07-40-07-dev-2cc217b7a", imageName: "ghcr.io/hyperledger/cacti-daml-all-in-one",

image

Cheers!

Rayn

@raynatopedrajeta I tried to publish the image but it fails to upload because one of the layers is 1.5GB and the upload keeps retrying to infinity.

Could you please reduce the layer size by splitting the commands in the dockerfile? I'm pretty sure it's this one:


# Download and install DAML SDK 2.9.3
RUN curl -L https://github.com/digital-asset/daml/releases/download/v2.9.3/daml-sdk-2.9.3-linux.tar.gz | tar -xz -C /opt && \
    cd /opt/sdk-2.9.3 && \
   ./install.sh && \
   rm -rf /opt/sdk-2.9.3/
petermetz commented 1 week ago

@raynatopedrajeta FYI: I pushed a small change to modify the readme of the DAML aio image so that it's easier to build the image in the future for publishing:

## Build an image locally

To build the daml-all-in-one image locally, use:
```sh
DOCKER_BUILDKIT=1 docker build \
  --file ./tools/docker/daml-all-in-one/Dockerfile \
  ./tools/docker/daml-all-in-one/ \
   --tag daio \
   --tag daml-all-in-one \
   --tag ghcr.io/hyperledger/daml-all-in-one:$(date +"%Y-%m-%dT%H-%M-%S" --utc)-dev-$(git rev-parse --short HEAD)


(Just saying this so you don't accidentally delete that change when you are resolving potential conflicts)
jagpreetsinghsasan commented 19 hours ago

@RafaelAPB here's the second PR (the first one got merged sometime back and we had dockerized daml all-in-one image created in that. This PR deals with the creation of DAML connector and test cases which tests the connector with the AIO)

petermetz commented 15 hours ago

@raynatopedrajeta Unfortunately I still can't publish the image because the problematic (oversized) layer is still 1.5GB and GHCR just won't have it at that size.

What I suggest is to refactor the image to have a multi-stage build where you call the daml sdk install in the first stage and then copy over the installed files in 3 or 4 chunks (or whatever makes sense) so that you can control each layer's size that way.

I'll paste for reference some of the commands I used while debugging the layer sizes so you can audit it for yourself as well:

The install folder that is 1.5GB (needs to be split across layers so that we can upload the image to GHCR)

#10 [ 6/17] RUN du -a -x / | sort -n -r | head -n 20
#10 1.070 2233840       /
#10 1.070 1452640       /root
#10 1.070 1452596       /root/.daml
#10 1.070 1452572       /root/.daml/sdk
#10 1.070 1452568       /root/.daml/sdk/2.9.3
#10 1.070 788968        /root/.daml/sdk/2.9.3/damlc
#10 1.070 709976        /usr
#10 1.070 603948        /usr/lib
#10 1.070 472316        /root/.daml/sdk/2.9.3/damlc/lib
#10 1.070 316644        /root/.daml/sdk/2.9.3/damlc/resources
#10 1.070 316644        /root/.daml/sdk/2.9.3/damlc/lib/resources
#10 1.070 292452        /usr/lib/jvm
#10 1.070 292440        /usr/lib/jvm/java-21-openjdk-amd64
#10 1.070 273036        /usr/lib/x86_64-linux-gnu
#10 1.070 225988        /root/.daml/sdk/2.9.3/canton
#10 1.070 225984        /root/.daml/sdk/2.9.3/canton/canton.jar
#10 1.070 207988        /usr/lib/jvm/java-21-openjdk-amd64/lib
#10 1.070 155492        /root/.daml/sdk/2.9.3/daml-sdk
#10 1.070 155456        /root/.daml/sdk/2.9.3/daml-sdk/daml-sdk.jar
#10 1.070 151100        /root/.daml/sdk/2.9.3/damlc/lib/damlc
#10 DONE 1.4s

Build command with progress=plain so that you can see the output of du when checking file sizes:

DOCKER_BUILDKIT=1 docker build --progress=plain   --file ./tools/docker/daml-all-in-one/Dockerfile   ./tools/docker/daml-all-in-one/    --tag daio    --tag daml-all-in-one    --tag ghcr.io/hyperledger/daml-all-in-one:$(date +"%Y-%m-%dT%H-%M-%S" --utc)-dev-$(git rev-parse --short HEAD)

An example of how to list the layers of any image with their sizes included (all layers must be below 500 MB otherwise we won't be able to reliably upload/pull them to/from GHCR)

#22 exporting to image
#22 exporting layers done
#22 writing image sha256:1b518c9931faccb21acba11e72827b74016516b9a5167693ccabf2e6ed32e408 done
#22 naming to docker.io/library/daio done
#22 naming to docker.io/library/daml-all-in-one done
#22 naming to ghcr.io/hyperledger/daml-all-in-one:2024-11-21T16-31-25-dev-b000e2a6a done
#22 DONE 0.0s

 1 warning found (use docker --debug to expand):
 - LegacyKeyValueFormat: "LABEL key=value" should be used instead of legacy "LABEL key value" format (line 3)
peter@zend:~/a/cacti-upstream$ docker history --no-trunc 1b518c9931faccb21acba11e72827b74016516b9a5167693ccabf2e6ed32e408
IMAGE                                                                     CREATED             CREATED BY                                                                                                                                                         SIZE      COMMENT
sha256:1b518c9931faccb21acba11e72827b74016516b9a5167693ccabf2e6ed32e408   About an hour ago   HEALTHCHECK &{["CMD-SHELL" "/quickstart/healthcheck.sh"] "30s" "1m0s" "1m40s" "0s" 'd'}                                                                            0B        buildkit.dockerfile.v0
<missing>                                                                 About an hour ago   RUN /bin/sh -c chmod +x /quickstart/healthcheck.sh # buildkit                                                                                                      190B      buildkit.dockerfile.v0
<missing>                                                                 About an hour ago   COPY healthcheck.sh /quickstart/healthcheck.sh # buildkit                                                                                                          190B      buildkit.dockerfile.v0
<missing>                                                                 About an hour ago   CMD ["--configuration" "/etc/supervisord.conf" "--nodaemon"]                                                                                                       0B        buildkit.dockerfile.v0
<missing>                                                                 About an hour ago   ENTRYPOINT ["/usr/bin/supervisord"]                                                                                                                                0B        buildkit.dockerfile.v0
<missing>                                                                 About an hour ago   EXPOSE map[7575/tcp:{}]                                                                                                                                            0B        buildkit.dockerfile.v0
<missing>                                                                 About an hour ago   EXPOSE map[9001/tcp:{}]                                                                                                                                            0B        buildkit.dockerfile.v0
<missing>                                                                 About an hour ago   COPY supervisord.conf /etc/supervisord.conf # buildkit                                                                                                             1.45kB    buildkit.dockerfile.v0
<missing>                                                                 About an hour ago   RUN /bin/sh -c mkdir -p /var/log/supervisor # buildkit                                                                                                             0B        buildkit.dockerfile.v0
<missing>                                                                 About an hour ago   RUN /bin/sh -c /quickstart/generate-jwt-token.sh # buildkit                                                                                                        217B      buildkit.dockerfile.v0
<missing>                                                                 About an hour ago   RUN /bin/sh -c chmod +x /quickstart/generate-jwt-token.sh # buildkit                                                                                               597B      buildkit.dockerfile.v0
<missing>                                                                 About an hour ago   COPY generate-jwt-token.sh /quickstart/generate-jwt-token.sh # buildkit                                                                                            597B      buildkit.dockerfile.v0
<missing>                                                                 About an hour ago   RUN /bin/sh -c echo '{"server": {"address": "0.0.0.0","port": 7575},"ledger-api": {"address": "0.0.0.0","port": 6865}}' > json-api-app.conf # buildkit             98B       buildkit.dockerfile.v0
<missing>                                                                 About an hour ago   WORKDIR /quickstart                                                                                                                                                0B        buildkit.dockerfile.v0
<missing>                                                                 About an hour ago   RUN /bin/sh -c daml new quickstart --template quickstart-java # buildkit                                                                                           32kB      buildkit.dockerfile.v0
<missing>                                                                 About an hour ago   ENV PATH=/root/.daml/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin                                                                              0B        buildkit.dockerfile.v0
<missing>                                                                 About an hour ago   RUN /bin/sh -c rm -rf /opt/sdk-2.9.3 /tmp/daml-sdk-2.9.3-linux.tar.gz # buildkit                                                                                   0B        buildkit.dockerfile.v0
<missing>                                                                 About an hour ago   RUN /bin/sh -c cd /opt/sdk-2.9.3 && ./install.sh # buildkit                                                                                                        1.48GB    buildkit.dockerfile.v0
<missing>                                                                 About an hour ago   RUN /bin/sh -c tar -xz -C /opt -f /tmp/daml-sdk-2.9.3-linux.tar.gz # buildkit                                                                                      1.16GB    buildkit.dockerfile.v0
<missing>                                                                 About an hour ago   RUN /bin/sh -c curl -L https://github.com/digital-asset/daml/releases/download/v2.9.3/daml-sdk-2.9.3-linux.tar.gz -o /tmp/daml-sdk-2.9.3-linux.tar.gz # buildkit   689MB     buildkit.dockerfile.v0
<missing>                                                                 6 days ago          RUN /bin/sh -c apt update &&     apt install --no-install-recommends curl openjdk-21-jdk supervisor openssl xxd -y # buildkit                                      676MB     buildkit.dockerfile.v0
<missing>                                                                 6 days ago          LABEL org.opencontainers.image.source== https://github.com/hyperledger/cacti                                                                                       0B        buildkit.dockerfile.v0
<missing>                                                                 2 months ago        /bin/sh -c #(nop)  CMD ["/bin/bash"]                                                                                                                               0B        
<missing>                                                                 2 months ago        /bin/sh -c #(nop) ADD file:ebe009f86035c175ba244badd298a2582914415cf62783d510eab3a311a5d4e1 in /                                                                   77.9MB    
<missing>                                                                 2 months ago        /bin/sh -c #(nop)  LABEL org.opencontainers.image.version=22.04                                                                                                    0B        
<missing>                                                                 2 months ago        /bin/sh -c #(nop)  LABEL org.opencontainers.image.ref.name=ubuntu                                                                                                  0B        
<missing>                                                                 2 months ago        /bin/sh -c #(nop)  ARG LAUNCHPAD_BUILD_ARCH                                                                                                                        0B        
<missing>                                                                 2 months ago        /bin/sh -c #(nop)  ARG RELEASE                                                                                                                                     0B

Notice this layer is the problematic one from the above excerpt:

RUN /bin/sh -c cd /opt/sdk-2.9.3 && ./install.sh # buildkit                                                                                                        1.48GB