At least `ghdl/synth:beta` and `formal` images are 14 days outdated

rodrigomelo9 commented 4 years ago

Hi. It seems that since https://github.com/ghdl/ghdl-yosys-plugin/issues/98, ghdl/synth:beta and ghdl/synth:formal has been not generated. Both of them are 14 days outdated (the change was 9 days ago), while the rest of ghdl/synth images were updated 15 hours ago.

eine commented 4 years ago

This sounds very strange to me. There is a delay for sure, because we don't have any cross-repo automatic trigger. However, ghdl/synth:beta should be updated twice a week, while most of ghdl/synth images are updated once every week. Hence, ghdl/synth:beta should never be older than others.

I will review the procedure, but I'll need some time to have it fixed, since I might need to force the regeneration of the sequence.

Meanwhile, you might want to execute https://github.com/ghdl/ghdl-yosys-plugin/blob/master/ci.sh locally, in order to have ghdl/synth:beta and ghdl/synth:formal built locally. Those use the latest available ghdl/pkg:buster-mcode to avoid rebuilding GHDL. However, you can uncomment L11-L20 if you want to have GHDL built too.

tgingold commented 4 years ago

The current ci is failing for ghdl-yosys-plugin. Apparently, yosys is not correctly installed.

eine commented 4 years ago

The current ci is failing for ghdl-yosys-plugin. Apparently, yosys is not correctly installed.

This is a separate issue because of a fix that was merged yesterday (YosysHQ/yosys#1883). Hence, the yosys version that is currently in all our images is outdated. It will be fixed on it's own between this night and tomorrow night (according to the regular schedule).

tgingold commented 4 years ago

Great!

rodrigomelo9 commented 4 years ago

Only FYI, uncomment L11-L20 and change --from seems not enough @eine:

Running hooks in /etc/ca-certificates/update.d...
done.
curl: (3) URL using bad/illegal format or missing URL

gzip: stdin: unexpected end of file
tar: Child returned status 1
tar: Error is not recoverable: exiting now
The command '/bin/sh -c apt-get update -qq && DEBIAN_FRONTEND=noninteractive apt-get -y install --no-install-recommends     ca-certificates curl && update-ca-certificates  && mkdir -p ghdl && cd ghdl  && curl -fsSL "" | tar xzf - --strip-components=1  && ./configure --enable-libghdl --enable-synth  && make all  && make DESTDIR=/opt/ghdl install' returned a non-zero code: 2

The problem is that GHDL_URL is empty. I used GHDL_URL="https://codeload.github.com/ghdl/ghdl/tar.gz/master" (there are several of red messages which scared my :P but seems not erros).

eine commented 4 years ago

@rodrigomelo9, let me prioritise this.

rodrigomelo9 commented 4 years ago

Yes, of course. Was only extra information. Sorry :P

eine commented 4 years ago

So, I triggered workflow cache (https://github.com/ghdl/docker/runs/577351641) and workflow synth (https://github.com/ghdl/docker/runs/577437846). This updated ghdl/synth:yosys, ghdl/cache:yosys-gnat, ghdl/synth:beta and ghdl/synth:formal (apart from ghdl/cache:gtkwave, ghdl/cache:formal and ghdl/synth:symbiyosys). Incidentally, I found a bug that was pushed to icestorm a few hours ago (cliffordwolf/icestorm#249). As a result, pnr and prog images are not updated yet: https://github.com/ghdl/docker/runs/577351652?check_suite_focus=true.

@rodrigomelo9, @tgingold, this should satisfy your needs (using up-to-date yosys, ghdl and ghdl-yosys-plugin; and having the CI of ghdl-yosys-plugin fixed). Note that it is not exactly the latest GHDL, tho. It was built 18h ago: https://github.com/ghdl/docker/actions?query=workflow%3Aghdl. Please, let me know if it works for you.

Only FYI, uncomment L11-L20 and change --from seems not enough

You are correct, that's not enough. Apart from uncommenting those lines, you need to somehow understand what's going on:

GHDL_URL is to download ghdl's sources. You can either set it or replace line 16 with git clone. The one you used is ok.
--enable-libghdl --enable-synth is not required in line 17. Indeed, I don't know if the later is supported, since it is the default now.
In line 28, you need to set --from=tmp, to use the image that you built when uncommenting the block above. Otherwise, it won't use the image that you just built.

Even if you had done that, ghdl/cache:yosys-gnat would have been outdated (as per Tristans comment above). However, to build ghdl/cache:yosys-gnat yourself, you would have needed another dockerfile: https://github.com/ghdl/docker/blob/master/dockerfiles/cache_yosys. At this point is where I decided to proritise it by manually triggering the builds.

Yes, of course. Was only extra information. Sorry :P

It's ok. As you see, the point was that it was easier for me to fix it than to explain you the details of the context. Of course, I'm open to answering any further question that you might have.

BTW, although not ready for everyone, there is this dot graph (https://github.com/ghdl/docker/blob/graph/.todo/graph.dot) which shows the relations between all the docker images that are maintained in this repo. Should you know how to improve the visualisation, I'd be glad to receive any help. I believe that making the diagram public, along with publishing https://github.com/ghdl/docker/tree/cli/cli, will make this repo more usable locally (such as you having a bunch of images updated without needing to understand the relations).

eine commented 4 years ago

cliffordwolf/icestorm#249 was merged, so I triggered pnr and prog images too: https://github.com/ghdl/docker/runs/577531180

rodrigomelo9 commented 4 years ago

BTW, although not ready for everyone, there is this dot graph

You right, it is not for everyone :P what about two graphics? Something like:

Left side

Right Side

There are a lot of dependencies (I am far to be an expert, but is not too much granularity?), so cli (with a README) will be welcome :P

I am already using the new ghdl/sunth:beta image, thanks :-D

eine commented 4 years ago

You right, it is not for everyone :P what about two graphics? Something like:

Yes. In fact, there are 4 graphics merged in a single one:

Images for testing GHDL on multiple distributions: workflow 'ghdl'.
Images for GHDL simulation, language server, VUnit, GtkWave, etc.: worklows 'vunit' and 'ext'.
Images for synthesis, pnr and prog, which are unrelated to GHDL: workflow 'cache'.
Images for synthesis which are related to GHDL: workflow 'synth'.

At some point in the future:

Would fit in https://github.com/eine/ghdl-packaging/ (renamed to ghdl/packaging).
Would belong here.
Are to be upstreamed to SymbiFlow.
Would belong here.

I believe that focusing on 3 will already make it much easier to understand. Your "second" graphic is precisely that content.

However, the dot file is generated programatically. Hence, the style/visualisation needs to be done programatically too. I've considered to generate a TikZ picture instead, because I think that positioning the elements might be easier. Unfortunately, the LaTeX engine is so heavy. A third alternative is to use some SVG lib for Python. Anyway, I cannot focus on the visualisation for now. It's mostly a tool for (understanding the) development.

There are a lot of dependencies (I am far to be an expert, but is not too much granularity?)

It is to be noted that not all of those items are actual images; some are intermediate build stages of the dockerfiles. Those could be published, but are not. The granularity in the 'cache' and 'synth' images is on purpose. The main motivation is to:

Build each tool from sources once, and only once.
Use the same base image and as many common packages as possible.

By doing so, we reduce the variability of the environment, at the same time that we optimise the size of each image and CI time for building and testing.

Regarding variability, if a user tests a design with ghdl/ghdl:buster-llvm-*, the same VHDL sources should work for ghdl --synth. If anything fails, only GHDL is to blame. By the same token, if something works on ghdl/synth:yosys but not on ghdl/synth:yosys-gnat, only GNAT is to blame. In ghdl/synth:beta, ghdl-yosys-plugin is the only additinal resource, etc. Each time we rebuild a tool or we duplicate the installation of a package, we are introducing a factor that can lead to hard-to-debug inconsistencies.

Regarding size, most of these images are expected to be used in CI environments without caching. This is because images are updated very frequently, so the benefits of caching are negligible. In this context, reducing the size of images is very important, as it slows down each execution. For this same reason, all the 'cache' and 'synth' images do share a single common parent and are stacked after each other. If you pull ghdl/synth:yosys and then want to pull ghdl/synth:beta, you will only retrieve the additional layers that correspond to GHDL and ghdl-yosys-plugin; but not a fully different image.

Overall, we are investing on build-system complexity to offer a better product and to be able to spot the source of bugs more easily. With "better" I mean to keep a consistency between the content of the images, so it is relatively easy to understand the differences (given the wide range of tools); and to not provide huge monolithic images (which is already available in many other sources).

Anyway, please, feel free to provide your thoughts. I know that many of these images are being used, but there is not much feedback, and we don't track the usage.

so cli (with a README) will be welcome :P

The CLI is not ready because not all the content has been converted from the current shell script. However, IIRC, all the 'cache' and 'synth' images are already supported. Precisely, the CLI is used in the CI workflows of that branch.

This is to say that I'd expect the help in the CLI to be self-explanatory:

# python cli.py 
================================================================================
                              ghdl/docker CLI tool
================================================================================
usage: cli.py [-q] [-v] [-d] {help,hub,build,task,graph} ...

Helper tool to build one or multiple images, to easily browse publicly available tags, and to generate 
graphs showing the dependencies between tags.

positional arguments:
  {help,hub,build,task,graph}
                        sub-command help
    help                Display help page(s) for the given command name.
    hub                 Show list of repos and tags at hub.docker.com/u/ghdl.
    build               Build a single image.
    task                Run a task, i.e. build a group of images.
    graph               Generate a dot graph from the multi-stage dockerfiles (WIP).

optional arguments:
  -q, --quiet           Reduce messages to a minimum.
  -v, --verbose         Print out detailed messages.
  -d, --debug           Enable debug mode.

Happy hacking!

So, instead of browsing hub.docker.com, you can use command 'hub' to get all the available images. There is a description of each group, and for each image the size, digest and update date are provided. For further info, the README of this repo tells the update frequency.

# python cli.py hub
Get 'run' info
Get 'build' info
Get 'ghdl' info
Get 'pkg' info
Get 'ext' info
Get 'vunit' info
Get 'synth' info
Get 'cache' info
Get 'dist' info

name      pull count  description
------  ------------  ------------------------------------------------------------------------------------------
run            37613  Images with runtime dependencies for GHDL
build          35078  Images with development depedendencies for GHDL
ghdl           19538  Ready-to-use images with GHDL and minimum runtime dependencies.
pkg            19363  Images with GHDL pre-built tarballs built in ghdl/build images
ext            10366  Ready-to-use images with GHDL and complements (ghdl-language-server, GtkWave, VUnit, etc.)
vunit           7413  Ready-to-use images with GHDL and VUnit
synth           4772  Ready-to-use images with tools for synthesis, P&R and formal verification
cache           2256  Images with external dependencies almost in the edge, but not part of GHDL.
dist             223

tag                 size (MB)    digest                                                            last updated
------------------  -----------  ----------------------------------------------------------------  ---------------------------
> run [23]
fedora30-mcode      116.89       a86d3e313da4c57c850d29fcc77579166a2d31ebdc3e6c9f88439a68c660e850  2020-04-01T00:13:13.229718Z
fedora30-llvm       149.44       10bc4c8a49e90b8d58be2279cf1c460d13c56af0aeb9586eea84f7db4efd9dae  2020-04-01T00:13:11.164838Z
fedora30-gcc        140.29       a02e328901d5c22889e9c79328b312cf973c881bc4922a5f75b411d4284cd97d  2020-04-01T00:13:01.504374Z
fedora31-mcode      112.98       228ddcdfa4a58d05359698a9ec92ab69b5aef3e753fe67abc74168dcc00afad0  2020-04-01T00:12:07.470152Z
fedora31-llvm       147.14       be6cafaff6d09881ee6137fa98798c6152d08168165e9adeb4e7d70976befc8c  2020-04-01T00:12:04.780292Z
fedora31-gcc        136.89       5faaf9d1b42b0fbef57408a89eeed69d6a0a7fdf3bd699e046513e05216b0aba  2020-04-01T00:11:36.702614Z
ls-ubuntu           106.14       45447b6ea56f2d5718daa810e647d2b8cb61425d525ad420bea7afd731f9773f  2020-04-01T00:09:33.277111Z
ls-debian           125.40       851fb88ea722ddfd3a344e90e954840b3dd28e36a9039fc1329546ffa0d69aa6  2020-04-01T00:07:13.063893Z
ubuntu18-mcode      67.71        a8bd773e6cb0c82c5702f74f4780e59bb900f6ec9d2ad6943787962372e3bbc4  2020-04-01T00:07:07.205431Z
ubuntu18-llvm-5.0   87.26        e1960832bb4ed83fe9638af76b81156adc8db28d604f79ebede7d92ddfdaf8a7  2020-04-01T00:07:05.17158Z
ubuntu18-gcc        77.79        ba0fc35bdd4d694c688a02ed8ef82b7ea848b3cfcc2e45c6b596e5a4b59cfb07  2020-04-01T00:06:56.11076Z
ubuntu16-mcode      79.48        897a72736f950da389effdb7357cd3cfabc7253e6838b28b2f1c84bece46a263  2020-04-01T00:04:40.215759Z
ubuntu16-llvm-3.9   95.70        a7c0006c58347939e5dbad0c9682efe8f012dd2ac6f92dd2eab243a989476f17  2020-04-01T00:04:36.357235Z
ubuntu16-gcc        89.30        b747440aa0395017268bcfb713dc042053abc3cd84b97b2349e382df2f7a7037  2020-04-01T00:04:29.653338Z
buster-mcode        68.03        cecd7dfb56f236b77594ca78ae108c3817914f4709d937e78c69866eb48b6766  2020-04-01T00:03:35.737032Z
buster-llvm-7       87.30        693528c559ba52671dba64d73e27a37ec0688c19c6ebc86b4cb525842976aa62  2020-04-01T00:03:33.549173Z
buster-gcc          78.76        401cc28770abc9d4afd8911bb2911c82e89ea46c201a50d1e05668efa3e52baa  2020-04-01T00:03:26.353024Z
sid-mcode           77.06        ddcd02c1c5ffe73de8675bfe249190466aea078a94bcb67ff41365ad5a297555  2020-03-16T00:04:21.104019Z
sid-llvm-8          97.42        0f8695b4c7a84abac538906c0c0116d74106339d7d9c06c8014eed3d77750766  2020-03-16T00:04:19.417908Z
sid-gcc             87.95        482324dd962f7fd6a7037fbd3a87a15ca26c9e6b57c799b110fef6d23623108d  2020-03-16T00:04:11.995048Z
fedora29-mcode      140.56       33839b3067b3fe70187d9e1b7194d8870ba6c174b471101eb749618a52a38748  2020-01-17T06:12:20.682239Z
fedora29-llvm       170.23       61995196f69923019a1b10f43983648d8967bd2bb163e96908afc1cd071d3926  2020-01-17T06:12:18.73263Z
fedora29-gcc        164.28       48acdbd1eacfdbc84dfc6e158e4b5da6feb998b1db900a879d536fe71dcefee2  2020-01-17T06:12:10.104486Z

> build [23]
fedora30-mcode      144.92       b815e779c6d904871bf770db75c950309d96201b5bab917d9f22c3696186aa2a  2020-04-01T00:12:47.598149Z
fedora30-llvm       233.15       03e403b8f7b664f8c8f216959bdc1b8bc8264f21e11fb23492963855959daa60  2020-04-01T00:12:45.732106Z
fedora30-gcc        183.92       ea2e4e627a97f5bb592fe523727565cc50e6a969474df49b64bc9e3d375d036e  2020-04-01T00:12:28.346914Z
fedora31-mcode      140.97       55755a04a745f614726f907191442596e89fabd3b810ee730cddaff1e94992f5  2020-04-01T00:11:22.164887Z
fedora31-llvm       250.28       c3a20ab37edebb264040b1addd32aa89f8f7d265420ee65fc9f86a48b671e30b  2020-04-01T00:11:20.066864Z
fedora31-gcc        180.22       578252591149875adc4186a1edad50652265f53b66cbaba4c8439d2a4c3ad2b1  2020-04-01T00:10:57.094919Z
ls-ubuntu           255.10       687d6d421bfb8de836e794728c4e41b1ecb5e829bf37f196eeec5e98e4b3d400  2020-04-01T00:09:25.705687Z
ls-debian           298.29       7a5ee52dc5acef87fffd7ffc000b192739e87bfe8afd090602dd7b1a57b61e6b  2020-04-01T00:07:03.866056Z
ubuntu18-mcode      96.92        56b3b5ac3e382d801e870c3e74215db4a342458d09db7fb616325d6a9853f0ef  2020-04-01T00:06:42.378711Z
ubuntu18-llvm-5.0   199.61       84c48b27253c8b85136a4e164c475edcdc3bb309362645e774c9988a4637b0a2  2020-04-01T00:06:39.889181Z
ubuntu18-gcc        145.94       7c752e28558cf20ee1fb90c1d4edb7ebdc8c2db52411d33b2998cde50714f9f8  2020-04-01T00:06:14.717688Z
ubuntu16-mcode      114.55       5d5c9f90fbd91341ffbffe9117d94271df253e57f6fc64125851c4f9aafdd478  2020-04-01T00:04:18.218535Z
ubuntu16-llvm-3.9   236.82       6288d4847c6a4abf5c8df02344740017c27e7f792ed4206ec1500df8fa166001  2020-04-01T00:04:15.651485Z
ubuntu16-gcc        164.43       b19decc3fc946445ca06a768f2a76a63b07f9de34f7162de3b3e600493f9287c  2020-04-01T00:03:48.323003Z
buster-mcode        96.24        23b2942edcf294b4e99eaf98a60a070cca30e88e434da33c3bd2a595d874e596  2020-04-01T00:03:14.732336Z
buster-llvm-7       196.78       8a5f7a2df3b261f45a7b4b1be25f809fc6bb36b3ad0cdc3733d0bfede6bf23aa  2020-04-01T00:03:12.867066Z
buster-gcc          145.09       88a23f8a9f0076faaa4b91e19c1d0d07b6032e72b208d0bed7ac3164b6370148  2020-04-01T00:02:48.08363Z
sid-mcode           132.92       0728c80f5a574a1405db8dce2dbba8037c227673fac888190ea818becebf4c60  2020-03-16T00:03:59.57796Z
sid-llvm-8          238.11       b759ca390a604bc2fdd635c43262240d28e5c1dfb647a554d5abfd841b156749  2020-03-16T00:03:57.742911Z
sid-gcc             187.69       083b775add824a5faec40e7b648ac098878c8d8bd95e5a4b3a239d834f843a5e  2020-03-16T00:03:31.014792Z
fedora29-mcode      169.05       419a538ffb618a38d020227c4d4ec4016e5ecf5ac2e90b95f4ff68cd0a83df63  2020-01-17T06:11:55.787964Z
fedora29-llvm       249.41       b04d5f1f2f33f3834f4b197b1f794ad76c3396cfb00324ea058b28bdd53cc5bb  2020-01-17T06:11:53.315878Z
fedora29-gcc        205.28       9b685b0c71c8304ec37fa27efcd5a6ffc04f0ec86cf252322fa29992fee14f7b  2020-01-17T06:11:35.160464Z

> ghdl [24]
fedora31-gcc-9.2.0  341.15       0ff6aa2533c32e6c20113f465cbcde1d461d8ddbe3515e2e83c897ffed8b07f4  2020-04-10T00:51:38.621847Z
fedora30-gcc-8.2.0  337.76       fc1cdfb86a4475b80443ec55b92870027e0a9100fb0d3b05eddbad0973058d98  2020-04-10T00:50:29.946499Z
sid-gcc-9.1.0       292.93       1161f9ee7f243d123335f7ebb14bc5911b135ddea1bad71b2c1a43d0f8801e36  2020-04-10T00:46:41.562742Z
fedora31-llvm       152.89       e29765707d060f5cd636175712fc62ada5562ed34aadb3473040645ef80122d5  2020-04-10T00:30:43.678264Z
buster-gcc-8.3.0    276.54       7b8e355fc33d8d0e73e4aedad123f8cec59d2d9cabbb4b644de53cec98b340bc  2020-04-10T00:24:49.959416Z
fedora30-llvm       155.19       cc63598b4beb012bdfb0aba306d8fddc250055c18b6d049ec2e5566bbdafcffc  2020-04-10T00:22:44.070969Z
ubuntu18-llvm-5.0   93.14        84648a44a19a44287a16b679b3cc3e71ae7e795be0d92c9f5161d5978f4f735c  2020-04-10T00:21:57.344139Z
ubuntu16-llvm-3.9   101.43       b84a0bb5cc36eb8e0f6ff04f6ce670a8f11ff4cfe077ed6f37c1903994234a1a  2020-04-10T00:16:25.681495Z
buster-llvm-7       93.19        1a2f541ed2b64ae5ebf6666f0ff882df980aa98b4f787b2a660c9620d120f6b7  2020-04-10T00:15:55.17358Z
fedora31-mcode      116.59       d5ea577c76c83d74c624ecc23f86bc659ce9a28e8ecc106d6dce8fb430c35de6  2020-04-10T00:09:00.808073Z
fedora30-mcode      120.49       58d8ac3587299893a6682e71fa29d655b3efb4e93291e8f831f91bb8aad445a7  2020-04-10T00:08:56.737485Z
ubuntu18-mcode      71.42        ae4cc229f43f981e0b8125a664ca55433a12c1f8ba8a06a0128aefaf55d93d4e  2020-04-10T00:08:23.596247Z
buster-mcode        71.74        d0f48ebcf9bcfdca137e6907926a9fce4e5459de0b8c8af705c9ae113e88f98c  2020-04-10T00:05:26.750524Z
sid-mcode           80.78        99e56bf34f4a830ef518bdb9a2e0be2cccb6d1bb31b35fb82e0efac660162c82  2020-04-10T00:05:24.056162Z
ubuntu16-mcode      83.06        aa7a2fc129a3acab2d0f1980b630ebf12afea1bcb24ec7f48b173b3838ff4cef  2020-04-10T00:04:25.788208Z
sid-mcode-gpl       80.47        22d414c9f9b964717b93e557154ec9adb9029b9a7e7bfa120fab7214fa9c9264  2020-04-10T00:04:09.374365Z
fedora29-mcode      144.09       764b4d880f91ddcd476a20d4db98e7b06ab083603a4b29530c212d681f1841e3  2020-01-17T07:50:40.075218Z
fedora29-gcc-8.1.0  362.79       60a2a6447c81734ef3ef3ae20c58c7096e87999e6071912392b72aca2115e4d4  2020-01-17T07:08:59.860764Z
fedora29-llvm       177.46       abee72aa7ff15c275137e999a136a8cb5927e1600f9f4c74e52e16851256b14f  2020-01-17T06:41:10.011076Z
buster-mcode-gpl    70.78        63d44e1629602917151047233408924405ca9d139c525e8d1ec0d94b5dfb6f5a  2020-01-17T00:03:33.321977Z
sid-llvm-8          95.99        077395fbfb72b1b85004bdc373b4b5da0a5df26605a7667f9b8a3f9197515374  2019-11-15T00:24:24.658395Z
fedora29-gcc-8.2.0  361.03       abb4c1a977c7e8f46c416437c77310e27a936101f19a740c744360cc799b8246  2019-09-16T06:22:42.38936Z
buster-gcc-7.4.0    265.95       47df84278daadfec43c0611f67ebd7d588b70073b703776e615b58392af67d71  2019-08-19T05:45:26.820473Z
buster-gcc-7.2.0    254.89       8b47bfddd826632235e843275b24b86e8e93119c5af8b2319d1d9241c25bbf61  2019-07-08T05:41:20.460358Z

> pkg [21]
fedora31-gcc-9.2.0  204.25       7c72b601e5aafcb3e1f0a9fdaef345fa6a35c0ae82e84869ab3d49c031d6f1c2  2020-04-10T00:52:26.937195Z
fedora30-gcc-8.2.0  197.47       98ab076fe3377661b77c0a173d1767d2f6441c972529fe73b14dc8ac16871e25  2020-04-10T00:51:24.149342Z
sid-gcc-9.1.0       204.98       0a9718945a7faf9e2ed8f69ede1d323472db33c6469d6faf839347148c26616a  2020-04-10T00:47:16.458957Z
fedora31-llvm       5.76         fa0ed595c9f73dbc02857f8780bb480a2dd0c162f4719a920f49a377a8c7cef3  2020-04-10T00:30:48.40952Z
buster-gcc-8.3.0    197.77       4f97de06c7170c59c2faa5c7e98af1edadcbf97ca7a3a383a24743dcff33d0dc  2020-04-10T00:25:22.958507Z
fedora30-llvm       5.75         9075560a6acbbfecd7c72b9ead401723d8f5333679525a6f6839cc20040c3f58  2020-04-10T00:22:48.299019Z
ubuntu18-llvm-5.0   5.87         9a1646aae29b98c4ee9331d35b02e7d026e778ba474cffa345825a089f2d3f2e  2020-04-10T00:22:05.318457Z
ubuntu16-llvm-3.9   5.72         326c4bb1691f23eeff43323ed54f76e9524a584e31acab3ccd0d6ee7056f4ea6  2020-04-10T00:16:31.077856Z
buster-llvm-7       5.89         ec3d1c74d48dd09317b2c6ba0797fa321ad12c711566a73ae616746a3f62c120  2020-04-10T00:15:59.393725Z
fedora31-mcode      3.61         c8b598b682e630e43e41d636fb078d9a05ce557aee817fcc0c4516b614adccab  2020-04-10T00:09:05.137823Z
fedora30-mcode      3.61         37385d32e9bd5599385b90e2ca8ee1aba7fdcb9ad6d531559dd78572a354252c  2020-04-10T00:09:01.116516Z
ubuntu18-mcode      3.70         f9bf468fca940ffa414faa20746cac1b0165485ab24a3bd0f2c25e017fc44554  2020-04-10T00:08:27.733709Z
buster-mcode        3.71         d329911796c5cd8c5b1e4c37274781c714a272a6d8653e2b2603558268c2657a  2020-04-10T00:05:30.341579Z
sid-mcode           3.71         fdfdde1627167048ce9bffbdf4b5b964c67d18851c8db647c87bd894dbb9db21  2020-04-10T00:05:27.91656Z
ubuntu16-mcode      3.58         606eb0cee45f9260836e796d0728eb1fda7a31112f246db45843701970f3ed14  2020-04-10T00:04:30.278644Z
sid-mcode-gpl       3.41         373f9e7a27d49f6445e8540f8c0d82b478714b9dff1635e9be8b7f1cd4336e3b  2020-04-10T00:04:13.661152Z
fedora29-mcode      3.53         7b99fbbb6c1c14efcf58f2353c759ca11008967b511200dd3bd9a208d44950f6  2020-01-17T07:50:44.420749Z
fedora29-gcc-8.1.0  198.50       5102245bb244cad06a7924bb46ffafd324fc3bc6f6cae8e00146af4b4de0b9d4  2020-01-17T07:09:38.641049Z
fedora29-llvm       7.23         8dfa3abed2c807fdfa422764d9304aa76345948830a2f06c2f2976f579fd9e56  2020-01-17T06:41:15.178728Z
buster-mcode-gpl    2.66         748216898b1e5092d550703a95be0f6601abc908db4ae86067dfeef9fcf687c0  2020-01-17T00:03:37.492202Z
sid-llvm-8          6.52         3ebbd25a028d000b9373436e5eb1bc21f78814b36d59b97960cee304bb664a49  2019-11-15T00:24:28.471309Z

> ext [5]
latest              266.01       35a076f14e55ed37baad606bf30937361b4e4a467e2eb2cdc64440f4338b0d4c  2020-04-08T00:10:58.840248Z
ls-vunit            183.76       cf1d65d196a889c9b623eb3b2870d6a9d09bc14574616cb1f0addb803ba9b11e  2020-04-08T00:11:01.618617Z
broadway            233.77       bcfff2c1a577696cae05a9d979b09ead9b820c4f688aacdc97a227c1b1e5bdfd  2020-04-08T00:10:40.401872Z
ls-debian           137.96       c8627957285009de3e02414c0d814221ba0bb171d023d6152baac8bb792b7317  2020-04-08T00:06:00.092934Z
ls-ubuntu           119.48       669ad02cfb1a2e1bd82a155108b6c416816a053b2f595a4573071e4cc7235b40  2020-04-08T00:05:54.863254Z

> vunit [6]
mcode               104.72       da923f2f80cd1b6a069acd0f3b4b83f7ed13840b44781c45602c74cd4c866590  2020-04-08T00:11:36.415547Z
mcode-master        138.19       6417e79dae71189812d4d68439c7f468dd63fb691c8f4ab097aa56449091c570  2020-04-08T00:11:31.545689Z
llvm                126.17       45db03c54652665aabc561cedcee5f817ed28d663d98beabaad3466ebc3508e6  2020-04-08T00:11:21.612176Z
llvm-master         159.64       7315388c7834f0ea455880d39299e6ed83cc20c5384de590b62f041e2c0099fa  2020-04-08T00:11:17.401271Z
gcc                 321.32       87479e8cdbadc598f66c5e69404337b021ae8bb0569d4b868cc771b17ed0b73f  2020-04-08T00:11:08.230775Z
gcc-master          354.79       5da852fece8657824a9b54d22c221cf0584b55a44a2f95a4896a6e8bc6ed2a09  2020-04-08T00:11:02.491315Z

> synth [11]
latest              73.93        3cbd380bf79e81f080a2e7ad9fbd767947de2f2a713b9b5fd552809f8fbea432  2020-01-20T20:06:21.793653Z
icestorm            55.25        c96a8e37934a2de43dea5d2f666a92dbce0462d33813e6959bfcde77c78181cf  2020-04-10T18:48:51.461797Z
trellis             246.49       0776615a047066fc2a7b80866752668e16a5946c3019c2c31522b0860db83c1c  2020-04-10T18:48:37.021177Z
prog                27.64        a8358c5d442d01c9abcfd9a04ce51e093d4e3834dc96ec85f1136677523b5cc0  2020-04-10T18:47:53.884694Z
nextpnr-ice40       327.43       ced9e91f858f61cd008e35c17e1fbaa03b0006bd92dca9955f1f7c7145da9a64  2020-04-10T18:47:50.154873Z
symbiyosys          213.73       ffe77674ef1fa8ce2a7063f8e03f7bbfbb2ff73141aa70d278681fb2736d1813  2020-04-10T18:47:32.747856Z
nextpnr-ecp5        282.53       164ff25982059720aad9beeb4192d490b45ba8fb910b52303e0825419b3b415c  2020-04-10T18:47:28.951651Z
nextpnr             378.06       2719b3b6169e787b838f97c8cdb01c7d873a61eba964d3217a0d29cb83ed4744  2020-04-10T18:46:59.390597Z
yosys               192.16       61510a8d0d99501f3a348f03c85f77bedb1205e36d2a3e51769c1637ba5b95e7  2020-04-10T18:46:48.396489Z
beta                197.77       bc6c87a727dca954236a5376c840adf56075955dbeb61f973720373882052723  2020-04-10T17:31:56.151374Z
formal              219.34       9889bd7924f431c70fdb9ccb5901ac6a26c33945408bfda440b7b6d732fed8eb  2020-04-10T17:31:53.945117Z

> cache [3]
yosys-gnat          193.69       91f5a5282a25b2f24c58af9896a858c9518fa58ef98cfba1fa27173b2d867446  2020-04-10T18:46:45.960555Z
formal              21.57        f1d84744fd8537a4d64d487c5faef92ab9258b03acb879e84215eb48bfa6150c  2020-04-10T18:36:03.0198Z
gtkwave             8.31         6b536bde1a55e60492169ae4be03b5aa86ca3069d3c0098f02a3902ab892d043  2020-04-10T18:13:26.064122Z

> dist [1]
rpm                 177.55       cc4c3e3c9757bbfe2bc23a9e3c6fc862e80c154a90dd2aa51461483fa4b92d04  2019-11-22T03:13:14.494746Z

If you want to rebuild an image (instead of pulling it), you just pick the group/repo name and the name of the image from the table above. For example, build synth yosys.

# python cli.py help build
================================================================================
                              ghdl/docker CLI tool
================================================================================
usage: cli.py build [-h] [--dry-run] <Repo> <Tag>

positional arguments:
  <Repo>      Repo - hub repo name
  <Tag>       Tag - target image tag

optional arguments:
  -h, --help  show this help message and exit
  --dry-run   Print build commands but do not execute them.

When you want to build not only an image, but a group (typically all the images in the direct dependecy chain), you execute a task. These are currently not explicitly documented (https://github.com/ghdl/docker/blob/cli/cli/tasks.py#L61-L83), but do match the "jobs" in the CI workflows.

# python cli.py help task
================================================================================
                              ghdl/docker CLI tool
================================================================================
usage: cli.py task [-h] [--dry-run] <Name> [<Args> [<Args> ...]]

positional arguments:
  <Name>      Name - unique identifier
  <Args>      Args - optional arguments/filters

optional arguments:
  -h, --help  show this help message and exit
  --dry-run   Print build commands but do not execute them.

Honestly, I would not know what to put in the README... I had thought about recording a screencast/gif showing the execution of the three commands above. Any suggestion is welcome!

I am already using the new ghdl/synth:beta image, thanks :-D

Great! I'm closing this issue, since it is solved. Nevertheless, please do continue de discussion.

rodrigomelo9 commented 4 years ago

@eine:

At some point in the future:

Would fit in https://github.com/eine/ghdl-packaging/ (renamed to ghdl/packaging).

Would belong here.

Are to be upstreamed to SymbiFlow.

Would belong here

So, seems a good idea to separate the graphics, for easier visualization. I know, dot can be tricky for positioning things. With some changes, I got:

SymbiflowGraph

Ok about the granularity :-D

Great about the self-explanatory help of cli.py. I think that how to get help on commands is not clear at all (the explanation when you run python cli.py was not enough when I read before your explanation). Maybe your explanation as a small tutorial (into the main REAMDE.md) could be used :-D a small item list with "if you want...".

Regards

rodrigomelo9 commented 4 years ago

Hi @eine

I was trying to use cli, with some problems. I know, it is an experimental branch, but could be useful to me :P

The first problem was that is not compatible with Python 3.5 (I am using Debian 9 XD). I installed python3.8 with pyenv.

Secondly, there was a small problem related with the dry_run parameter:

diff --git a/cli/cli.py b/cli/cli.py
index f0cbfd6..05720ec 100755
--- a/cli/cli.py
+++ b/cli/cli.py
@@ -47,8 +47,8 @@ class Tool():
         task(name, args, dry_run)

     @staticmethod
-    def build(repo, tag='latest'):
-        build(repo, tag)
+    def build(repo, tag='latest', dry_run=False):
+        build(repo, tag, dry_run)

Do you like a PR for that?

And now:

$ python3 docker/cli/cli.py build synth beta
BUILD: synth beta
::group::[DOCKER build] synth : beta
CMD: docker build -t ghdl/synth:beta --target=beta - < /..../repos/external/docker/dockerfiles/cache_pnr
Sending build context to Docker daemon  6.144kB
Error response from daemon: failed to reach build target beta in Dockerfile
Traceback (most recent call last):
  File "docker/cli/cli.py", line 127, in <module>
    CLI().Run()
  File "docker/cli/cli.py", line 80, in Run
    ArgParseMixin.Run(self)
  File "/..../.pyenv/versions/3.8.2/lib/python3.8/site-packages/pyAttributes/ArgParseAttributes.py", line 297, in Run
    self._ParseArguments()
  File "/..../.pyenv/versions/3.8.2/lib/python3.8/site-packages/pyAttributes/ArgParseAttributes.py", line 309, in _ParseArguments
    self._RouteToHandler(args)
  File "/..../.pyenv/versions/3.8.2/lib/python3.8/site-packages/pyAttributes/ArgParseAttributes.py", line 313, in _RouteToHandler
    args.func(self, args)
  File "docker/cli/cli.py", line 112, in HandleBuild
    self.build(args.Repo, args.Tag, args.dry_run)
  File "docker/cli/cli.py", line 51, in build
    build(repo, tag, dry_run)
  File "/..../repos/external/docker/cli/tasks.py", line 173, in build
    build_image(*sw[repo](tag), dry_run=dry_run)
  File "/..../repos/external/docker/cli/build.py", line 45, in build_image
    do_build(cmd, fptr)
  File "/..../repos/external/docker/cli/build.py", line 33, in do_build
    check_call(cmd, stdin=stdin)
  File "/..../.pyenv/versions/3.8.2/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['docker', 'build', '-t', 'ghdl/synth:beta', '--target=beta', '-']' returned non-zero exit status 1.

Maybe, it is extremely easy for you to solve that and I could use it :P

ghdl / docker

At least `ghdl/synth:beta` and `formal` images are 14 days outdated #27