ngPlant on macOS with latest Homebrew packages and some extra goodies

[GwynethLlewelyn]

Hello there,

Recently, I was asked to evaluate many different tree-generation applications, and, naturally enough, I was referred to ngPlant as a good, cross-platform solution that was still in use by many designers (albeit not being actively maintained...). For some unexplained reason, I did get the source code, but didn't even check to see if there was a pre-compiled binary available for macOS (there was!... and it worked and all).

So, assuming that I would not need to change much to get it compiled, I rolled up my virtual sleeves and started doing changes.

It was very quickly clear that this would not be a simple, one-hour task. I thus made a fork and started hacking the code, intending to submit it 'later on'. As you can see, it took me way longer than I intended to spend on it, but, alas, I hate leaving things 'half-done' and then abandon it, so I put all my spare time into getting ngPlant to compile under my Mac, running macOS Big Sur Beta 11.5 with Homebrew installed.

That wasn't an easy task, mostly because of the following reasons:

1. I'm not a professional programmer. Professionally, I'm just a humble, low-end, old-school system manager for small organisations; academically, I'm doing a PhD on things related to virtual worlds and artificial intelligence, so I do have decades of solid experience in programming — just not at the professional level. Thus, please take that into account when reviewing whatever code I've typed: I'm sure it's not written in the best idiomatic way and could benefit from a lot of cleaning up.
2. I've never used SCons before. I wasn't even aware it existed before I started on this fork. I've been strictly a Makefile girl, back in my university and post-university days — that was three decades ago. I've toyed with nant just for two projects in Mono/C#, but that's pretty much all I had ever played with. I'm aware that there are more choices (Ninja, for instance) but, in truth, I don't know anything about those. I'm not really even what some would call a Makefile witch — I just know the basics, which have suited me well for years. Thus, delving deep into SCons was a completely novel experience for me; as a consequence, I'm sure I took way too much time in trying to understand the basics and why some things would work and others baffled me completely. In the end, I got it to do what I needed it to do (get a fully compiled version that actually runs without immediately crashing and even does something useful). It required lots of nasty hacks, some of which are very likely anathema for experienced SCons users. These will most definitely require a lot of revision, to make sure that it complies with the established practice of using SCons. I've also learned to hate it through and through, especially because, in the end, I didn't need to change a single line of the C++ code; all the 'trouble' I had was due to the SCons configuration. Or, in other words, the tool that was supposed to help a full compilation to complete successfully and thus save time for the developer... that was the 'weak link' that took hours to master and finally conquer. The C++ code, by itself, was fine.
3. One of the reasons for all my issues with SCons is simple to explain: I'm not a Python programmer. In fact, in the past, I've spent æons fixing Python code to get it to run. There must be something conceptually 'wrong' with Python at its core, because I tend to always have problems when requiring to fix Python code. It seems that Python is the kind of language that is inherently broken from scratch and that you spend all the time trying to fix it so that it does what it supposes to do. Then again, perhaps it's me — I might simply not be correctly wired to think in Python. I could go on and on, but I'd say that Python would be in my Top Ten List of Never-To-Use Programming Languages. Alas, Python is ubiquitous these days — you can't avoid stumbling upon it everywhere. It might be argued that if I loved Python, I'd like SCons as well, and, more important, I would at least understand better why certain things didn't work — some were pure Python issues; others were quirks of SCons; but, to me, they seemed to be one and the same. So, I only blame myself for the time needed to fix all those things, often with very ugly hacks that will make a professional Python developer shudder and scream in pain. Alas! Sic transit gloria mundi, or, more modernly, valar morgulis.
4. The core of ngPlant uses C++ (and sometimes some C, too). Great. That was actually a programming language I learned at university. I like C. I even used to like C++ (back in 1989, though, not the syntax monster it became). I felt I was on more solid ground here. Still, it's true that I haven't written anything 'serious' in C/C++ for decades; even its remote cousin, C#, which I also hate with a passion, is something that I haven't touched since, oh, 2009 or 2010 or so. And don't get me started on Java. As a consequence, while I still like C, and can tolerate the current versions of C++, I'm quite rusty at either. Well, let me rephrase that: I'm very rusty. So rusty, in fact, that I had to look up on Wikipedia if C++ had a garbage collector or not, because I couldn't remember. And there are lots of things that have been popular idioms in C++ for the past decade or two which I'm not in the least familiar with. Thus, I was expecting that my lack of current, working experience in C/C++ would be a huge drawback — but, to my astonishment (or perhaps not...), there was not a single line of code that needed a change. Sure, I get a lot of deprecation warnings, but I can live with those, so long as it compiles; and that certainly worked (also, most of the warnings came from wxWidgets, not ngPlant). See, that is one of the reasons I still like C: it was invented when I was born, in 1969; and you can still compile it (well... within limits) and it still works!
5. However, there was a snafu with the ngPlant code: it relies on embedded Lua for some of its magic scripting abilities. As you might guess, I'm pretty clueless about Lua, too (at this point, you might be wondering what I usually write code in). But I also haven't got anything against it — I know that it is employed successfully on a trillion projects for exactly the same reason that ngPlant uses it: extraordinary scripting abilities in a fast, performant platform, that can get very easily embedded on any kind of application, with bindings for practically all programming languages out there. Therefore, I have a certain positive 'prejudice' towards Lua: someone who picked it to provide scripting abilities inside their application is the kind of programmer I love — someone who does not reinvent the wheel, but, instead, picks something ready-made and widely used to accomplish what they need to do. So, I thought, so far so good. Sure, I don't know even the basics of Lua, but I'm sure that this is something I could tackle if there was any need. Alas, in the case of Lua, the main issue was that ngPlant requires a very specific version of Lua to work. It seems to work well with Lua 5.0 and 5.1, but nothing before that — and most certainly nothing after it. I have 5.4.3 installed on my Mac: that's what Homebrew installs. It seems that I could, to a degree, 'port' the Lua bindings from 5.2 to 5.4.3... but 5.1 is simply too old to work. I noticed that I would have to literally make changes on some twenty or so C++ files, each of which required completely different Lua bindings, and would need very careful testing and debugging to get anything to even compile (much less work!). I couldn't afford that amount of time. Fortunately, you've included the full source code of Lua 5.1, perfectly tailored for working with ngPlant. Great, it was just a question to force the compilation to use the internal Lua library and ignore whatever was installed on my own system! Ok, so, probably Lua 5.1 has more loopholes and security flaws than a Russian-hacked SolarWinds device. But the move to a more recent version needs to be (re)written by a qualified professional programmer, not a quick-and-dirty hacker like me.

That said, I managed everything to work under macOS Big Sur 11.5 Beta and Homebrew 3.1.12 (or thereabouts). In other words: everything but Lua comes from standard packages as deployed by Homebrew, and is thus kept up to date. This is not only true of things like libpng or libjpeg, but also wgWidgets (and Scons itself!). To get it all to work nicely, all I needed was to add a config/macos.py file and pre-fill what was needed. Oh, sure, there was a lot of hacking on the myriad files employed by the SCons ecosystem, but I tried to make those changes as small as possible (and fencing them off when they were clearly Mac-specific, so as not to disturb compilation under Linux or even Windows). Again, as said, I'm pretty sure that a lot of the changes can be made in much more elegant ways (for example, at some point, I was having a problem with a lambda function that would add the relative paths to a list of .cpp files to feed them to the compiler — so I commented it out and just added the relative paths directly on the list. Ugly, sure, but it took me far less time than debugging what's wrong with that particular lambda function...).

I think that much of the trouble I had here and there was with the Python version. That's not really surprising; I'm always having issues with Python versions, or, worse, Python library versions. In other words, while anything before Python 3 has been deprecated, Python 2.7 is still popular (macOS Big Sur still comes with 2.7.16 as the 'standard'). However, most of the files used by SCons have been written for something below Python 2.5 — when huge changes were introduced (as far as I understand, the idea was to get Python 2 programmers more used to the way Python 3 works, and thus a lot of typically pythonesque idioms of early Python 2.X version were abandoned in favour of things more consistent with Python 3). Now, I have no opinion about what version of Python — 2 or 3 — is 'best' or 'more appropriate'; that's a discussion I am not qualified to participate in; in fact, I'm absolutely agnostic in that issue, I hate them both equally 😄

Whew. That was a mouthful. You can see that I love to write 😉 — especially ranting, of course.

As a bonus to all the changes to get this compiled, I've also pulled the old documentation pages on SourceForge and pushed them to the GitHub wiki (harder to do than I thought — because, while the original files on SourceForge are in Markdown, I do not have access to them, just to the rendered HTML). For some reason, it seems that you can't fork Wikis related to a project if the project being forked had no Wiki to start with — or at least that's what happened — so I have put those on a separate repository: https://github.com/GwynethLlewelyn/ngplant/wiki

It's essentially the same as on SourceForge, with a bit of cleaning up, and, of course, completely new instructions to compile ngPlant under recent versions of macOS (I kept the old ones too, for historical purposes). You're most welcome to copy them all over to the 'official' project, assuming that this is easy enough to do. If it's not that easy, I might be able to turn over the ownership of the Wiki repository to someone else; or, worst-case scenario, I think I can zip all those files and send it to you guys (you'd need to copy them manually, I guess).

I think that's all for now. Do whatever you please with this pull request; if you have decided never to incorporate any of it, that's fine with me, it suited my purposes (getting a compiled version on macOS!), and now that I'm finished with all this work, I was informed that my team picked up a different tree-generator application... lol! That means that I'm not available to do further cleaning up; which is a pity, because my next task was to make sure if everything I did still worked under Linux, and see how hard it would be to compile it all in my Synology NAS, running a modified Linux version on an ARM 64-bit processor.

Cheers, have a wonderful weekend, and thanks so much for all the work put on ngPlant!

[stager13]

Hi Gwyneth!

Thank you very much for your work on getting ngplant sources to be compatible with the latest MacOS environment and detailed description on how you did it. Believe me, I know it wasn't that simple. Let me explain why some decision regarding frameworks/ languages were made and also my thoughts on all of this:

1. First of all, when you say you're not professional programmer I think you're wrong :) - Several decades in software development showed me that not leaving things half-way done and continuing to work on solving complex (and sometimes quite boring) problems is much more important than the knowledge of many languages, patterns etc. So from this point of view what you accomplished and how you deal with tasks is quite professional in my opinion. And also I don't think that someone who solved so many issues on all levels (python/scons/wxwidgets/c++//MacOS-releated etc) should be recognized just as an "amateur" programmer :)

2. I used to use SCons at that time (when I started ngplant development) because it looked much easier than writing Makefiles by hand (I had experience with autoconf/automake but wasn't sure if it is suitable to use in cross-platform build with wxWidgets etc). And cmake wasn't as stable as I wanted at that time. I think that if I have to start ngplant development now I'd choose cmake. Huge problem with scons is that for simple projects it works like a charm, but as soon as your projects need some auto-configuration it became quite problematic (maybe I just don't use it against it's ideology :) )

3. Regarding scons/python. I think that the main problem is not python language or scons themselves. I like python, but the way python developers handled python2->python3 transition caused many problems (at least for me). Oh, and it's ok if you don't like it - everyone has personal preferences and there is nothing wrong with it.

4. Yeah, C is still the best language to make sure that code is portable (language is not enough to ensure portability, but at least it doesn't stay on the way of the programmer). It has it drawbacks (as any other language) but this is what we have :)

5. I didn't look at what changed in Lua for a last couple of years. Definitely need to look there.

Ok, it was really interesting to read about your journey with ngplant. After reading your comment I tried to build ngplant in Ubuntu 20.04 and found that it doesn't compiles there (I think because of the similar problems you encountered with scons/python). So what I will do, I will go through changes you've made and incorporate them. Also I will check and add changes needed to build it on Ubuntu 20.04. Can't promise it will be fast, but I will do it for sure!

BTW, I'm not sure if you're aware about existence of web version of ngplant at http://ngplant.org/webplant/webplant.html . It doesn't support Lua plugins and g-mesh branches, but it allows to edit models and save them locally in .nga and .obj formats. And you can run it in browser :)

Thanks again for you efforts!

[GwynethLlewelyn]

Hiya @stager13!

Truth to be told, I wanted to see how things would work under Ubuntu 20.04 as well — and on the Synology NAS, too. The theory is that Ubuntu 20 is the 'standard' Linux distribution these days, for several reasons; it's also one of the easiest to maintain, and probably that's the biggest reason why it's so popular — and that was my reason for installing it on my tiny little server 😸 Alas, because it's a server (a 'headless' server, as it's known these days...), there would be some issues in getting it to display anything — although I guess that ngpview, which seems to use X11, could be tested, by opening an X11 session on my Mac (currently, it crashes with a Segmentation Fault on the Mac, but... perhaps I can get it to work under Ubuntu).

Compiling things on the Synology NAS is one of my personal hobbies. Synology clearly states that their systems are not meant for development purposes, and always recommend cross-compiling applications if one really needs to install anything outside the built-in package manager (which is the only recommended way of installing anything).

Well, I love challenges, and my theory is: if you can make it in New York, you can make it anywhere... I mean: if you can compile/run it on an underpowered ARM64 Linux server, you can get it to compile/run on anything else, including, say, an old Android phone or a jailbroken iPhone 😆 One may ask what would be the point of running ngPlant on a jailbroken iPhone (or a NAS...) — I'd say, none whatsoever except having fun! lol — and proving that it is possible.

Anyway, thanks for your very kind words, but I'm really not a pro in the usual sense, that is, someone that is paid to write 50 lines of code per minute, or whatever metric is popular these days. I'm more the kind of person who writes one line of code per hour — because, well, I have to look up so many things I don't know that I take ages until I get something to actually work 😏 On the other hand, it's true that I've been often asked to deal with legacy code because I'm stubborn enough to keep hacking at until it works!

As for SCons, I guess that it must have some merits, or it had been discontinued long ago. It just feels weird to me to have a very complex framework, written in a completely 'foreign' high-level language — Python — just to 'dynamically' compile a different, high-level language. That means learning Python even if you wish to develop exclusively in C/C++... granted, using any other building tool also requires learning its syntax, and even if you had done it with CMake, I would have to read its documentation lol — as said, I hardly know the basics of plain old make, and a few things of the nant tool. However, I totally agree with you that going with CMake would be a better choice these days, especially because people would be able to choose among several generators — Mac users could use XCode, Windows users could use Visual Studio, and Linux users in general could either go with good old Makefiles or something more sophisticated (and faster!), such as ninja; you wouldn't need to worry about these details! So, sure, by all means, replace SCons with CMake on a future version of ngPlant!

In any case, I'm spoiled, these days I'm a [Go](https://golang.org] girl, mostly because I thought that a 'new' programming language (actually, it's well over a decade old) designed by Ken Thompson, 50 years after C, would 'fix' all the problems of contemporary high-level languages. Perhaps surprisingly, it does! And there is even a wxWindows wrapper for Go...

Newsflash: one of the issues I had (mentioned on the wiki — don't forget to get it as well, it seems that it's somehow a 'separate' project, for reasons unknown to me) was corrected today! Through a convoluted process, someone managed to get in touch with the Homebrew maintainers and implement a simple fix. This was an issue about symbolic links being wrongly created on the Mac version of wxWidget (a very stupid — and easy-to-fix — bug). This now works, so I've also updated the wiki and deleted the paragraph that mentioned that issue.

So the only thing needed to compile ngPlant on the Mac is the interface to a more recent version of Lua!

Post-scriptum

1. As mentioned, during compilation, there is a certain amount of warnings; fortunately, none are fatal, and they do not interfere with the compiled app when it finishes compilation/linking. Most are related to deprecated constructs when invoking the old, internal Lua library; I expect that, if you fix that in the future, most (if not all!) of those warnings will disappear.

   However, to get it working under the Mac, I had to bump up the required macOS version to be at least 10.9. Today I tried a clean recompile again — trying out the 'new' version of wxWidgets, to be sure that it didn't break anything! — and, although it certainly worked well, it also gives a warning that compiling for 10.9 is 'not recommended'; rather, the recommendation is to set it to compile for 10.15 and above. My guess is that this is something that Apple is pushing to all developers, even those outside the Mac ecosystem (i.e. App store, Mac developers, etc.) in order to make sure that recently recompiled apps are already future-proof, now that Apple is rolling out their new Macs with Apple Silicon CPUs (they're ARM64). This is just a speculation on my side. I really have no idea how many people are still using super-old macOS versions such as 10.9; I can guess that there will be a few, though, because very old hardware will still run, even after a decade, but it means that it won't run the latest versions of macOS (mostly because Apple artificially disallows them to do so; you can hack this check, but the results are, at best, unstable... newer versions of macOS rely on newer chips and built-in functionality, and, if those are absent, it will fall back to software emulation of those features — which may be impossibly slow...).

   In other words: it is possible that if you get ngPlant working with the latest and greatest versions of the embedded Lua library, there will be no choice but to have it compiled for macOS 10.15 or above — unless someone wishes to do everything manually (as opposed to use the versions packaged via Homebrew), by using earlier versions of the libraries, i. e. following the 'old' instructions.

2. No, I wasn't aware of the Web version! Now I feel stupid; it has everything that my team needed (they most certainly don't know Lua, neither want to learn it...), and, as a bonus, it has something really hard to find — a very reasonably convincing eucalyptus tree!! Arrgggh now I feel even more stupid! I could have had so much less trouble...

   On the other hand, it would also mean you'd never get an updated version lol — so I guess the effort was not in vain!

3. If you get to review the whole app in the future, consider supporting trackpads on the interface! 🤣 Currently, when using a trackpad, all I can do is zoom in/out and do a bit of rotating around, but that's it. Remember, we mouse-impaired Mac users are still used to just have one button! Sure, there are key combinations to 'emulate' the other two buttons, as well as 'gestures' on the trackpad that work like the scrolling wheels... but at some point, it becomes too complex to figure out the combinations (we have just 10 fingers, and we need 2-3 for gestures on the trackpad...). I do use several 3D platforms successfully with just the trackpad — and I'm not ashamed to admit that I'm pretty good at figuring out all the complex combination of pressed keys and trackpad gestures — but I have to say that ngPlant's own interface eluded me completely. I guess I could just attach a mouse and see what works (and eventually remember what particular key/button combinations are necessary to do the same on the trackpad), but I'm too lazy to figure it out 🙈

Anyway, thanks for all the work put on ngPlant, I really appreciate it!

[stager13]

Hi,

I added a new branch named 'macos-compatibility' specially for the changes related to the macosx. Would you please change the base branch of this pull request from 'master' to 'macos-compatibility'? I'm not sure if it's possible without creating new pull request (I'm not very familiar with pull-request workflow). Please let me know if it is possible. I prefer to do this not in master branch to make sure that new changes will not accidentally break something. After all work on this be finished I'll merge it back into the main branch.

PS: I did some work on Ubuntu 20.04/python3 compatibility, hope I will have time to merge and check everything soon. Unfortunately I don't have access to the latest MacOS X so I will try to be very careful to not break your fixes :)

[GwynethLlewelyn]

Sorry about only replying today...

I'm also not sure how to do it on my side — except for submitting a new PR, of course — although it seems that you can change it, according to these instructions: https://docs.github.com/en/github/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/changing-the-base-branch-of-a-pull-request

I have no idea if this will work! If it doesn't, let me know, I'll be reading git manuals until I figure out a solution 😉

[stager13]

Hi,

We're all busy with other tasks, so there is no need to sorry! I've merged you pull request to the macos-compatibility branch and will continue to work on merging things into the main branch.

Thanks!