Linux install instructions

[Robinlovelace]

I'm up for helping out with this. Exciting stuff.

[fiftysevendegreesofrad]

Oh hello! I've never yet tried to compile this on Linux. I suspect a small amount of work is needed to get gcc to compile it, and I'm not sure how dlls and ctypes work there either.

[Robinlovelace]

Coincidentally, I'm in Hereford right now. Will message you. I know some bright people we can get on the case - good candidate for a Linux/GCC/C++ minihack (I'm thinking of Andrew Smith here).

[fiftysevendegreesofrad]

Actually compiling may not be necessary - first step just copy over the install dir and see if the python scripts work.

On Sat, 1 Jun 2019, 09:07 Robin, notifications@github.com wrote:

Coincidentally, I'm in Hereford right now. Will message you. I know some bright people we can get on the case - good candidate for a Linux/GCC/C++ minihack (I'm thinking of Andrew Smith https://github.com/virgesmith/ here).

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/fiftysevendegreesofrad/sdna_open/issues/1?email_source=notifications&email_token=AC7WKTO4JUMQ3XLVEVOTCY3PYIU2DA5CNFSM4HR3FDRKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWW3N6Q#issuecomment-497923834, or mute the thread https://github.com/notifications/unsubscribe-auth/AC7WKTJCSMCCORHMMNUUAETPYIU2DANCNFSM4HR3FDRA .

[Robinlovelace]

Heads-up: discussed with Python/C++/Linux pro @virgesmith. One for a minihack Andrew?

[fiftysevendegreesofrad]

Ok I tried my own previous suggestion. It doesn't work as dlls don't run on Linux.

Two options

1. use WINE to run a windows python installation on Linux. Copy sDNA's windows install dir and start using it under WINE.

or

2. port the relevant parts of the code and recompile with GCC. This is of course the long term solution. For starters, sdna/sdna, sdna/geos and sdna/muparser are all needed. The latter two presumably come with linux support. The former is code for windows DLL and needs compiling instead to an shared object .so file. Then sdnapy.py will need to be modified to load this instead of the dll.

[Robinlovelace]

Thanks for the update @fiftysevendegreesofrad, I think putting all energy into the 2nd is the way forward. I hate WINE (or more subtly, going directly for the long term solution should be more time efficient).

[Robinlovelace]

Any idea of the type of skills that are needed to support this? It's beyond my ken but keen to help support this, e.g. by doing some kind of 'community call out' if that would help.

[fiftysevendegreesofrad]

Compiling c++ shared libraries (.so) on linux, and calling shared libraries using Python CTYPES on linux. If you know of people willing to help that would be fantastic! Shouldn't be a huge job for someone who has done those things before.

[Robinlovelace]

Is this up your street @virgesmith? Sounds like a problem matched to your Python/C++/Linux expertise.

[virgesmith]

if you want it to work on all platforms but only accessible via python I'd make it into a python package, https://github.com/virgesmith/humanleague would be a good starting point

[Robinlovelace]

Many thanks for quick reply Andrew, sounds sensible.

[fiftysevendegreesofrad]

I imagine Robin is keen on an R package as well! But in either case we need to continue supporting use via most of the existing means (QGIS, ArcGIS, command line - autocad less crucial) certainly there should be a way to achieve this from a single source tree.

[Robinlovelace]

certainly there should be a way to achieve this from a single source tree.

Too true. Next step I imagine is to actually get it running on Linux so it's at least partly platform agnostic. Are there any build tools you'd recommend @virgesmith ? And what about an Rcpp interface (I imagine that would be doable but a bit outside my ken again).

[JamesParrott]

Hi everyone. Compiling sDNA for Linux has been at the back of my mind for a while. I think I've got a subset of sDNA to compile on Ubuntu in gcc!

However I absolutely do not claim to have gotten it working. There are over 8000 lines of compiler errors to fix. So over to you guys. It's been fun, but unfortunately, I strongly suspect supporting Linux is too much work to be worth it.

https://github.com/JamesParrott/sdna_lite/releases

To recreate it you'll need to either uncomment the final RUN command in the Dockerfile, or run this in the Docker container yourself (and run the container using compile_in_container.sh):

g++ -shared -o lib_sDNA_ubuntu.so -fPIC sDNA/sdna_vs2008/*.cpp sDNA/sdna_vs2008/*.h sDNA/sdna_vs2008/IteratorTypeErasure/any_iterator/any_iterator.hpp sDNA/muparser/drop/src/*.cpp sDNA/muparser/drop/include/*.h

Geos is compiled too in a separate build stage, and its artefacts copied into the image, but I don't think the above command links it - gcc probably also needs output/arm/capi/geos_c.h and output/arm/lib/libgeos_c.so or similar.

[Robinlovelace]

Sounds like we need some hardcore C++/Linux programming support. If you guys cannot fix it I don't know who can!

[JamesParrott]

Absolutely. I forgot to mention this was done on an ARM server, so the .so won't run on your x86 or aarch64 machines anyway. My goal was to do compilation for different platforms in CI, but I'm not going to be persuing that now!

The Python ctypes calls might need a bit of tweaking too, to use the .so instead of sdna_vs2008.dll, but that's straightforward.

[Robinlovelace]

I thought Linux was easier for development of software :man_shrugging:

[JamesParrott]

It is. A fresh Windows 11 machine now, won't even let users install software outside of the MS app store by default, let alone develop it effectively.

The challenge here is different - making an OS specific program, support another OS, or even OS agnostic. The majority of the code in sDNA is straightforward, and will run in Linux just fine. The tricky part is unpicking the occasional Visual Studio-specific library and Windows header, understanding what they're doing, and replacing them with cross platform versions, or switching them out for Linux equivalents, without breaking something on either platform.

[JamesParrott]

@Robinlovelace Here's a version for Ubuntu Linux that mostly works, bar a handful of diffs from the Windows test data and a seg fault:

https://github.com/fiftysevendegreesofrad/sdna_plus/actions/runs/9584489142

An external R installation is required (with car and optparse) for sDNA Learn and Predict . If there's interest, the seg fault needs ironing out. This would allow dynamic import of Geos, enabling sDNA Prepare. I can get rid of the Windows R-portable too with an if statement in the CMakeLists.txt.

[Robinlovelace]

Awesome, thanks @JamesParrott !

[Robinlovelace]

P.s. respect for using the upcoming systems programming language / "C replacement" zig in the workflows! https://github.com/fiftysevendegreesofrad/sdna_plus/blob/main/.github/workflows/compile_dll_with_zig.yml

[JamesParrott]

Cheers! I don't think I'd be allowed to add Zig code to sDNA, and I don't know much Zig yet anyway.

Nonetheless, even though I didn't work out how to link the MS Windows C libraries with it (or compile Python) - so there will be bugs aplenty with those builds, overall I would agree with the claim that the Zig compiler is a drop in C++ compiler.

It's basically Clang, but a lot faster to get going with (both are LLVM under the hood, like Rust), and compared to 5-7GB for Visual Studio (or even MSYS2 and MingW64, or the Rust Windows build chain), at only 80MB Zig's an incredibly useful tool.

[Robinlovelace]

Yeah, Rust seemed like the ideal low level language, until I saw compile times! Not that I'm qualified to comment: all low level languages are baffling to me (and Rust is the most likely useful for my work with great GeoRust community).

In any case, in terms of this issue, do you want me to be a guinea pig and test the new instructions and report back as a user?

[JamesParrott]

Rust compile times have got a lot better. I'm sold on the time trade off, as it turns so many run time errors into compile time bugs. There's a great deal to like about it. But it's opinionated.

It's early days with this issue, but if you have time, that would be super helpful! Please do.

I wondered if there was interest in sDNA from Mac users (I don't think it would be much work to build for Mac now). And if the multi-threading works, I'm assuming Linux cloud servers are a lot cheaper per core than Windows ones.

[Robinlovelace]

I'll probably hold fire for now, I'm a long time Linux user (mostly Ubuntu on various devices) and thinking it may be a better strategy to wait until the optimum time to test things. Feel free to tag me when that time approaches.