search

awslabs / palace

3D finite element solver for computational electromagnetics
https://awslabs.github.io/palace/dev
Apache License 2.0
233 stars 49 forks source link

Question: why not Julia? #63

Closed smartalecH closed 1 year ago

smartalecH commented 1 year ago

First off, this is fantastic work. Well done, and thanks for sharing!

I see many examples are launched via Julia. I'm wondering why the whole package wasn't written in Julia? As I'm sure you know, there are some rather sophisticated parallel FEM assemblers/solvers in Julia (eg Gridap) plus you benefit from the extreme composability that Julia offers (source-to-source autodiff, seamless gpu support, etc).

I'm sure there's a good reason... and I'd love to hear it 😀

sebastiangrimberg commented 1 year ago

Hi Alec, thanks for your interest in Palace! Great to hear from you.

This is a good question and indeed one which came up and was addressed in the early days of this project. As you mention, Gridap provides a great ecosystem for finite element methods and solvers in Julia. It is rapidly maturing, but there are certain features which are missing/in development and make it a less desirable alternative to other finite element libraries for the case of Palace. For example, when starting development Palace in early 2021 GridapDistributed.jl was in its very early days (v0.1.0 was released in Sept. 2021) and offered only basic distributed memory functionality. Another example is that Gridap only very recently added support for H(curl)-conforming (Nédélec) elements which are common in electromagnetics applications on meshes which consist of non tensor-product element topologies. It's unclear if mixed meshes or prism/pyramid elements are supported yet.

We built Palace instead on the MFEM library which had proven success in many large-scale FE codes. MFEM is a C++ library, and the language question of whether to write Palace in C++ as well versus write in a higher level language and write bindings to the parts of the MFEM API which we need was also considered. Palace was always designed to be a standalone application for users versus a library with a more extensive API, and thus the choice of language had different implications. At the end of the day, we wanted to provide an interface for end users to easily build + run simulations with Palace, which is possible programmatically via high-level languages as the examples show without a library interface. A lot of high-performance scientific software like the libraries used by Palace is written in C or C++ and thus it made the most sense to write Palace in a language like C++ too where we could easily develop alongside these third-party libraries and make use of the active research advancements being implemented in these codes. GPU support, for example, is a not so straightforward topic when you consider the aspects of multi-GPU or different GPU vendors, and so even in Julia you are bound to find situations which are not so "seamless." There a lot of active work going into building high-performance and heterogeneous/portable simulation codes and Palace is poised to take a advantage of a lot of this with sufficient abstraction even though it isn't written in a language like Julia.

I hope this answers your question and you agree with the reasoning. Please let me know if there are pain points you see in your desired use cases for Palace and we can discuss further how to go about resolving them!

smartalecH commented 1 year ago

Hi Sebastian, thank you for the very thorough response! I now understand why you pursued this route, and I think you made the right choice 😁 .

Are you interested in shipping a jll (e.g. using BinaryBuilder)? You can then use something like Clang.jl to generate wrappers if you wanted (although I'm sure you are already aware of this). If you are interested, I can pull something together (once I get a bit of free time...)

Thanks again!

sebastiangrimberg commented 1 year ago

I've used BinaryBuilder in the past, but maybe you can help me understand a bit more here. If I understand correctly, I would think the benefit of shipping Palace as a JLL via BinaryBuilder would basically be that you could add it to a Project.toml file for a Julia project which allows users to download a prebuilt binary of Palace very easily. Then, users could use the provided ExecutableProduct to run Palace as documented in https://docs.binarybuilder.org/stable/jll/#ExecutableProduct.

Is that the intention? This is definitely an interesting distribution mechanism especially for supporting a large range of platforms, but I have to say I am a bit hesitant about maintaining such a feature. It seems more flexible to support standard build and distribution systems for people to install Palace as they please and then use it in Julia by having it on their path and using run(`palace ...`). It seems like we don't need to ship a JLL/use BinaryBuilder really because we aren't trying to expose any LibraryProduct objects or anything Julia specific.

sebastiangrimberg commented 1 year ago

I will close this for now, please feel free to re-open and follow-up if you have any related Q's!