rems-project / sail

Sail architecture definition language
Other
621 stars 114 forks source link

issues when packaging on Debian #453

Open yuzibo opened 9 months ago

yuzibo commented 9 months ago

Hi,

I am trying to package sail to Debian. But I got a very strange issue:

E: Error: unit Base64__Unsafe exported in sail-plugins but already exported by libbase64-ocaml-dev/libbase64-ocaml v3.5.1-2+b1
E: Error running /usr/bin/ocaml-md5sums  -v --md5sums-dir debian/libsail-ocaml-backend-ocaml-dev//var/lib/ocaml/md5sums --md5sums-dir debian/libsail-manifest-ocaml-dev//var/lib/ocaml/md5sums --md5sums-dir debian/libsail-output-ocaml-dev//var/lib/ocaml/md5sums --md5sums-dir debian/libsail-smt-backend-ocaml-dev//var/lib/ocaml/md5sums --md5sums-dir debian/libsail-lem-backend-ocaml-dev//var/lib/ocaml/md5sums --md5sums-dir debian/libsail-latex-backend-ocaml-dev//var/lib/ocaml/md5sums --md5sums-dir debian/libsail-doc-backend-ocaml-dev//var/lib/ocaml/md5sums --md5sums-dir debian/libsail-c-backend-ocaml-dev//var/lib/ocaml/md5sums --md5sums-dir debian/libsail-ocaml-dev//var/lib/ocaml/md5sums --md5sums-dir debian/libsail-coq-backend-ocaml-dev//var/lib/ocaml/md5sums --md5sums-dir debian/libsail-sv-backend-ocaml-dev//var/lib/ocaml/md5sums  --package sail-plugins dep < debian/sail-plugins.olist.debhelper at /usr/bin/dh_ocaml line 501.
make: *** [debian/rules:9: binary] Error 255
dpkg-buildpackage: error: debian/rules binary subprocess returned exit status 2
(sid-amd64-sbuild)root@dev:/home/vimer/build/rfs/ocaml/package/7_sail/sail-0.17.1# find . -name *.cmxs
./debian/tmp/usr/lib/ocaml/libsail/libsail.cmxs
./debian/tmp/usr/share/libsail/plugins/sail_plugin_lem.cmxs
./debian/tmp/usr/share/libsail/plugins/sail_plugin_sv.cmxs
./debian/tmp/usr/share/libsail/plugins/sail_plugin_ocaml.cmxs
./debian/tmp/usr/share/libsail/plugins/sail_plugin_latex.cmxs
./debian/tmp/usr/share/libsail/plugins/sail_plugin_c.cmxs
./debian/tmp/usr/share/libsail/plugins/sail_plugin_output.cmxs
./debian/tmp/usr/share/libsail/plugins/sail_plugin_coq.cmxs
./debian/tmp/usr/share/libsail/plugins/sail_plugin_doc.cmxs
./debian/tmp/usr/share/libsail/plugins/sail_plugin_smt.cmxs

Now I just have a little understand about ocaml package on Debian, so please correct me if I am wrong. On Debian for sail, we can use:

dh $@ --with ocaml --buildsystem ocaml_dune

To build it. But there is one check called ocaml-md5sums but it was to check dependency :(.

If the error is true, so maybe we need to improve here. If not, then we close it.

yuzibo commented 9 months ago

Grep more code snapshot:

(sid-amd64-sbuild)root@dev:/home/vimer/build/rfs/ocaml/package/7_sail/sail-0.17.1# grep -snri "Base64" .
./debian/control:13: libbase64-ocaml-dev,
./sail_doc_backend.opam:24:  "base64" {>= "3.1.0"}
./sail_ocaml_backend.opam:24:  "base64" {>= "3.1.0"}
./src/sail_doc_backend/docinfo.mli:82:type embedding = Plain | Base64
./src/sail_doc_backend/dune:6: (libraries libsail omd yojson base64)
./src/sail_doc_backend/dune:7: (embed_in_plugin_libraries omd base64))
./src/sail_doc_backend/docinfo.ml:108:type embedding = Plain | Base64
./src/sail_doc_backend/docinfo.ml:110:let embedding_string = function Plain -> "plain" | Base64 -> "base64"
./src/sail_doc_backend/docinfo.ml:393:  let encode str = match Config.embedding_mode with Some Plain | None -> str | Some Base64 -> Base64.encode_string str
./src/sail_doc_backend/docinfo.ml:395:  let embedding_format () = match Config.embedding_mode with Some Plain | None -> Plain | Some Base64 -> Base64
./src/sail_doc_backend/sail_plugin_doc.ml:80:  | Some "base64" -> Some Docinfo.Base64
./src/sail_doc_backend/sail_plugin_doc.ml:82:      Printf.eprintf "Unknown embedding type %s for -doc_embed, allowed values are 'plain' or 'base64'\n" embedding;
./src/sail_doc_backend/sail_plugin_doc.ml:97:      "<plain|base64> Embed all documentation contents into the documentation bundle rather than referencing it"
./src/sail_ocaml_backend/dune:14: (libraries libsail base64)
./src/sail_ocaml_backend/dune:15: (embed_in_plugin_libraries base64))
./src/sail_ocaml_backend/sail_plugin_ocaml.ml:183:  |> Base64.encode_string |> output_string f;
Alasdair commented 9 months ago

I don't know about debian packaging, but each Sail backend is built as a dynamically loaded plugin. Libraries those plugins need are embedded within them. It looks like maybe your ocaml-md5sums tool doesn't understand that, and thinks the plugin cmxs are regular libraries somehow?

Alasdair commented 9 months ago

You should be aware also that the current build setup assumes that either A) Sail is being installed by opam, or B) the source repository is available somewhere on the user's machine (because they are building it with dune themselves). This is so the backend plugins and library files can be found automatically at runtime.

If you are packing and either of those two assumptions are not true, then there might need to be changes to this repository to acommodate that.

yuzibo commented 9 months ago

Thanks for quickly reply.

It looks like maybe your ocaml-md5sums tool doesn't understand that, and thinks the plugin cmxs are regular libraries somehow?

I think so here.

You should be aware also that the current build setup assumes that either A) Sail is being installed by opam, or B) the source repository is available somewhere on the user's machine (because they are building it with dune themselves). This is so the backend plugins and library files can be found automatically at runtime.

The background is we want to package sail-riscv to Debian so we have to package sail here first. In this way, we have to get rid of opam to build OCaml package like sail here. In sail itself, I can build it with Debian dune build system and I got these Debian packages inspired by sail/libsail/sail_* except these plugins.

libsail-c-backend-ocaml-dev_0.17.1-1_amd64.deb       
 libsail-ocaml-dev_0.17.1-1_amd64.deb
libsail-coq-backend-ocaml-dev_0.17.1-1_amd64.deb     
 libsail-ocaml-dev-dbgsym_0.17.1-1_amd64.deb
libsail-doc-backend-ocaml-dev_0.17.1-1_amd64.deb     
libsail-latex-backend-ocaml-dev_0.17.1-1_amd64.deb   
libsail-lem-backend-ocaml-dev_0.17.1-1_amd64.deb    
libsail-manifest-ocaml-dev_0.17.1-1_amd64.deb       
libsail-manifest-ocaml-dev-dbgsym_0.17.1-1_amd64.deb  
 libsail-output-ocaml-dev_0.17.1-1_amd64.deb
 libsail-smt-backend-ocaml-dev_0.17.1-1_amd64.deb
  sail-ocaml_0.17.1-1_amd64.deb

sail-ocaml-dbgsym_0.17.1-1_amd64.deb
libsail-ocaml-backend-ocaml-dev_0.17.1-1_amd64.deb

I tried many time to put these those plugins under any packages or anywhere I can try to skip the ocaml-md5sum check but failed. I think you are right. There is one issue for ocaml-md5sum tool maybe.

I should report it to Debian OCaml team but it is not time for now. we have to package some dependencies first before this. Once done, we can report it to maintainer of dh-ocaml and to investigast it.

yuzibo commented 9 months ago

I forget to mention it here: sail-riscv will need these plugins during build.

yuzibo commented 1 month ago

Update:

I am almost close to bringing sail to Debian now. Only waiting for linksem to be uploaded.

But the situation is a bit complicated when I prepare to package sail again carefully. The issue is how to deal with those library plugins. For example, we can see one plugin with one backend as example:

vimer@host:~/build/rfs/ocaml/package/for-debian/7_sail/github/sail$ opam show --list-file sail_lem_backend
/home/vimer/.opam/default/doc/sail_lem_backend
/home/vimer/.opam/default/doc/sail_lem_backend/CHANGELOG.md
/home/vimer/.opam/default/doc/sail_lem_backend/LICENSE
/home/vimer/.opam/default/doc/sail_lem_backend/README.md
/home/vimer/.opam/default/lib/sail_lem_backend
/home/vimer/.opam/default/lib/sail_lem_backend/META
/home/vimer/.opam/default/lib/sail_lem_backend/dune-package
/home/vimer/.opam/default/lib/sail_lem_backend/opam
/home/vimer/.opam/default/share/libsail/plugins/sail_plugin_lem.cmxs

if putting the plugins into /usr/share/libsail/plugins/sail_plugin_lem.cmxs in deb package, the lintian will report:

E: libsail-ocaml-dev: arch-dependent-file-in-usr-share [usr/share/libsail/plugins/sail_plugin_lem.cmxs]
# Here Debian means arch-dependent file should be put into `/usr/lib/x86_64-linux-gnu/ocaml/5.2.0/sail_lem_backend/share/

From my simple understanding about this, these plugin files were library files also, but I remembered you mentioned these plugin cmxs files are not regular libraries. To fix the issue, we have two methods:,

  1. Use lintian-override to override the error, but this needs reasonable explanations, so could you help me to understand why we put these plugin into .../share/libsail/plugins/sail_plugin_lem.cmxs for opam system? this will not only to convince Debian reviewer to accept this override but also to fix the original issue, which I disbale to check it from ocaml-md5sums.

  2. We can did some adjust for the repo on Debian packaging as you suggested, I was trying it but it failed. For example, if I package sail-riscv with sail Debian package I have built on my local env, but the sail-riscv will report:

sail -dno_cast -c_preserve _set_Misa_C -O -Oconstant_fold -memo_z3 -c -c_include riscv_prelude.h -c_i[54/1949]
scv_platform.h -c_no_main model/prelude.sail model/riscv_xlen64.sail model/riscv_flen_D.sail model/riscv_vlen.
sail model/prelude_mem_metadata.sail model/prelude_mem.sail model/riscv_types_common.sail model/riscv_types_ex
t.sail model/riscv_types.sail model/riscv_vmem_types.sail model/riscv_reg_type.sail model/riscv_freg_type.sail
...
scv_insts_vext_red.sail model/riscv_jalr_seq.sail model/riscv_insts_end.sail model/riscv_step_common.sail mode
l/riscv_step_ext.sail model/riscv_decode_ext.sail model/riscv_fetch.sail model/riscv_step.sail model/riscv_ana
lysis.sail model/main.sail -o generated_definitions/c/riscv_model_RV64
Error:
/usr/share/libsail/plugins: No such file or directory
make[2]: *** [Makefile:257: generated_definitions/c/riscv_model_RV64.c] Error 1

So could you tell me if I should adjust where/scripts to tell Sail to find these plugin files correctly at runtime if put these plugin files into like /usr/lib/x86_64-linux-gnu/ocaml/5.2.0/libsail/plugins. Please ignore the multi-arch path here I can deal with on Debian side.

Thank you very much and sorry If I have too many questions for you.:)