Registerd binary wrapper and julia package

[mloubout]

Hi

Since we [SLIM]() work a lot in julia and some external user could be interested, I have created a julia native wrapper for rsf which consist of:

• RSF_jll a Julia binary library with rsf libs and programmed precompiled so that it can be added and installed in a second. I was only able to configure it for a couple of architecture but would like to extend it. PR at https://github.com/JuliaPackaging/Yggdrasil/pull/6627
• Madagascar.jl which is basically a lightweight version of m8r.jl . still few bugs to fix but mostly there.

So was just wondering if it's something you are fine with and/or interested in.

Cheers

[deckerla]

Love this idea To elaborate, having Madagascar.jl as a registered Julia package would make package management / dependency issues significantly simpler if we were doing deployment of Madagascar functionality on cloud instances. I haven't really tried that with m8r.jl. It may work well, but I just have limited knowledge & experience using the Julia Madagascar wrapping. Using Madagascar.jl it should also be straightforward to package Madagascar operators as Jets and use them within the COFII framework.

[mloubout]

The good news is that it seems to pass all the test nicely on ubuntu so that's a great start. The main thing is that it would basically avoid the massive work to compile it and have a nice portable framework with all the libs and executables.

Yes wrapping it in Jets/JOLI/LinearMaps/.... would be a really nice extra as well.

Waiting for the jll to be merged and I'll register the initial POC version of Madagascar in general registry once I fxed the docstrings.

Binaries

RSF_jll Madagascar.jl

[sfomel]

Please compare with the old interface to show how this improves it https://reproducibility.org/blog/2015/09/28/julia/

The wave propagation example https://github.com/ahay/src/blob/master/api/julia/test/afdm.jl is tested in book/rsf/rsf/afdm https://github.com/ahay/src/blob/master/book/rsf/rsf/afdm/SConstruct

[mloubout]

The interface is the same the difference is that the actual libraries and executable are wrapped into Julia binaries so that it can be installed directly from Julia as a standardized Julia package

It's also automatically tested so I'll make sure to add those

[deckerla]

This means that using the already compiled Madagascar libraries is as simple as

using Pkg

Pkg.add(“Madagascar”)

Downside in relation to m8r.jl is that if the underlying c code changes the _jll have to be recompiled, so it works better for stable code rather than development.

That said, it makes using established Madagascar utilities significantly easier and faster than installing Madagascar regularly.

Sent from my iPhoneOn Apr 24, 2023, at 5:45 PM, Mathias Louboutin @.***> wrote: The interface is the same the difference is that the actual libraries and executable are wrapped into Julia binaries so that it can be installed directly from Julia as a standardized Julia package

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: @.***>

[mloubout]

Ok so Madagascar_jll) is now registered, i,e the precompiled sources only

Madagascar.jl now setup with it a tested with CI including the afdm example above revamped for the new interface so you can just run julia afdm.jl. (Test results there)

Working on adding support for additional platform but looks like a good start and will probably register it fairly soon. Probably gonna compute some reference values for afdm to check correct solution as well.

It can be simply installed as

] add  https://github.com/mloubout/Madagascar.jl

on a linux system and won't need any other installation steps

[tmasthay]

It appears to me that while the CI tests pass for Madagascar.jl, the wrapper for sf_floatwrite will dump the binary content into the terminal.

See the Run tests workflow at this link to see this behavior. It is a fork with a single empty commit to reproduce the latest CI run (last one was deleted due to GitHub storage timeout). You will have to scroll down to line 416 of the output to see the beginning of the binary dump.

I suspect that the issue lies in the wrapper for sf_floatwrite. If I simply comment out the writing to the file when running julia afdm.jl, then the binary dumping does not occur (of course, neither does writing to an rsf file). I have also adding some debugging info at this link for reference. The same behavior takes place in run_tests.jl for me.

Finally, I would just like to note that at each iteration of the time-stepping loop of afdm.jl, the length of the binary dump gets larger and larger, so it does seem to be writing the information correctly. Furthermore, on the first iteration, the header information of the rsf is printed to stdout as well.

[mloubout]

Looks like afdm.jl was setup with stdout output ('Fo = Madagascar.sf_output("out")'). I have changed it to a file to make sure it doesn't blow up the log.

You can see here the log is now fine

[tmasthay]

@mloubout I am trying to build a GitHub Action that checks against the executables listed in Madagascar_jll so that a pull request can be sent to Yggdrasil for regular, automated releases of the precompiled binaries. When I build Madagascar on my Pop!_OS Ubuntu 22.04 system, my build system generates 176 executables that are not in the build_tarballs.jl script, and for some reason, I also do not build two that are in build_tarballs.jl. I know you did this over a year ago at this point, but do you recall why the subset of ExecutableProduct commands was chosen?

Attached here is a workflow that shows how I built Madagascar from source (what I believe to be essentially a carbon copy of the workflow from build_tarballs.jl) and then a simple script that does string processing for comparison.

Some executables listed like sfnpyCorr I imagine have pretty clear reason such as sfnpyCorr which I imagine depends on numpy and thus doesn't make sense to include in this julia package.

[mloubout]

Hi @tmasthay . Thanks for reaching out.

As you pointed out I haven't really looked at this in a bit, there is PR opened I need to finish on Yggdrasil that add supports for more architectures as well you might wanna look at.

For the list of executable, I don't remember how I generated that list, I think I made a basic bash script that read the content of the bin dir but might have ran it on a broken install so I wouldn't be surprised if there is missing ones in the current released version. Ideally that by-hand list should be replaced by a searchdir for more robust lib discovery. That's what Madagascar.jl does to generate the wrapper for all the executables dynamically.

I'll see if I can find some time next week to cleanup the PR and if there is extra details I can share.

[tmasthay]

@mloubout I talked with @sfomel, and we decided it might be better to support a build from source as well. I have posted a pull request on main of your branch that supports from source with a detailed message (see TODO at the bottom...I think there is a way to support both precompiled and source builds by figuring out compile-time environment).

https://github.com/mloubout/Madagascar.jl/pull/2

TLDR on the pull request: to do from source, use main branch (since rsf_read and rsf_write are pure julia and thus would be "easier" than precompiled binaries), and the precompiled branch HEAD is the same as the current HEAD of your repo.