search

svaarala / duktape

Duktape - embeddable Javascript engine with a focus on portability and compact footprint
MIT License
5.94k stars 514 forks source link

Rewrite repo Makefile into CMake #246

Closed svaarala closed 7 years ago

svaarala commented 9 years ago

The current repo Makefile is large and quite messy and only works on Windows through Cygwin. Rewrite the Makefile into a cleaner CMake version piece by piece, and eventually remove the current Makefile. The main motivation is to make it easier for users on Linux, OSX, and Windows to modify Duktape, create distributables, run tests, etc. CMake has been requested by multiple users.

A few activities that conflict with this a bit:

Notes:

There's some discussion in #241.

svaarala commented 9 years ago

Current targets categorized roughly:

Binaries and libraries, other distributables:

Code policy etc:

Testing:

Website:

Distribution:

Download / clone:

Documentation:

Misc:

svaarala commented 9 years ago

As a bit of background, early on I actually used SCons to manage Duktape building. SCons handles variants reasonably well and tracks changed files etc well. But it's a bit difficult to maintain so I moved to a flat simple Makefile.

Duktape Makefile builds several Duktape binaries from the same sources but with different compiler and Duktape options (i.e. variants). I'm not very familiar with CMake, I'll have to read up on how CMake handles variant builds or multiple build environments (compilers, compiler options, etc).

JoshEngebretson commented 9 years ago

CMake can be rough going on the frontend, especially if your first build with it is quite complex. CMake's killer feature is the quality of the output, not the CMake setup "language" which takes some getting used to

svaarala commented 9 years ago

@JoshEngebretson Any suggestions for a multi-platform alternative? CMake seems to be popular for Windows users compared to using e.g. Cygwin (which I personally do on Windows because I'm a Linux user normally :-).

JoshEngebretson commented 9 years ago

I would definitely use CMake. It has some warts (they all do), but it is kind of the gold standard right now. I can just relate that the very first thing you try with some new thing always seems to be the most complicated you can conceive :)

fatcerberus commented 9 years ago

Re: SCons: I've found that it works very well for Unix builds (including OSX) but is a pain in the rear to get working on Windows. MSVC's build process is so much different from everything else that 9 times out of 10 it's easier just to open the MSVC IDE than to try to build anything on the command line in Windows.

kphillisjr commented 9 years ago

I have tried various build engines and I have found that options are extremely limited for cross-platform support to begin with.

kphillisjr commented 9 years ago

I started the processs of converting the build scripts over. I have also made a quick pull request for this code with pull #278.

markand commented 8 years ago

I'll take it.

svaarala commented 8 years ago

Related issue about Python versions: https://github.com/svaarala/duktape/pull/646

The situation is quite different at the moment, so it'd be important to figure out what the makefile should and should not try to do:

svaarala commented 7 years ago

The current dist/configure tools are pure Python2 and there are no Makefile dependencies to create an end-user distributable or configure Duktape for compilation in the repo checkout. The only remaining step is compiling the C files.

Since there's no activity on this issue and there's been relatively little interest in that final compilation step, closing this issue.

If there's a concrete change proposal, please open a new pull request :)

rcdailey commented 7 years ago

I'm starting on this effort (please reopen). I just want to get you to review your initial posts and let me know if those requirements are still accurate. I'm still learning how your build process works. An overview for newcomers hoping to contribute would be useful (if there's already a document for this, please point me to it).

So far in the 5 minutes I've dug into this, it seems like files in src-input are not compiled verbatim. Amalgamated header and source look to be generated, and that output gets compiled. Is this accurate? Your python scripts are complex and it will take some time to understand everything.

My first goal is to get the duktape targets building first, with working installation rules. After that I'll focus on your custom targets (the static analysis tools and such). The long term goal is to keep everything as CMake-centric as possible, this means:

svaarala commented 7 years ago

I'm not sure I agree with all the goals above, e.g. rewriting the test runners to be CMake-centric would be a step down because they're now Python utilities that can be executed independently. Also Duktape repo Makefile doesn't (and shouldn't IMO) support installation, because the main point is just to create the distributable and compilation/installation is very target specific (remember Duktape targets include platforms like Amiga, MS-DOS, etc).

So before you start major work, it's probably best to discuss the scope a little more. I wouldn't personally spend too much time porting the whole development Makefile into CMake, as many of the targets are rarely needed even when modifying Duktape.

rcdailey commented 7 years ago

they're now Python utilities that can be executed independently

Could you explain this a bit for me? Independently meaning you can run the tests in parallel?

Keeping things self-contained in CMake will reduce the external dependencies required to build, test, and deploy your library which has huge savings. Of course, I'd love to discuss these in more detail with you because I do not have the domain knowledge you do about the build system requirements for your library. I'm approaching things at a basic level right now.

Note that I am not converting anything verbatim. I'm starting the CMake scripts from scratch with only the essential/required features. I use the Makefile as reference material for understanding how to build the C libraries (which translation units to build, preprocessor definitions, include directories, etc).

Definitely let's talk about scope. I'd like to approach things in baby steps if possible. Right now I'm just working on libduktape getting built on multiple platforms. You do have some configuration of duktape.h.in that requires access to Git and things, which I consider optional, and I am not going into that level of detail just yet.

I won't comment on what the Makefile should/shouldn't do because I am unfamiliar with make-based build scripts, but CMake installing headers, libraries, shared files (like docs) and such makes absolute sense to me. This is done independently per platform you support. I'm happy to go in-depth with you on this. One of my requirements as a user of your library is the ability to install your library to my SYSROOT so that I can use your library as an external dependency.

Let me know how you'd like to discuss this. We can discuss here, or I'm happy to move to email. Just let me know what works best for you.

svaarala commented 7 years ago

Regarding source compilation, the raw sources in src-input cannot be compiled as is for several reasons:

These steps can't realistically be handled by a Makefile (or CMake build files) and Python is so far the most practical tool for them: it's more easily installed and maintained than Node.js for example. Self hosting (= building the sources using Duktape-driven tools) would also be viable, but Python is more straightforward. The only real downside is the eventual Python2 to Python3 transition which is painful, but that is pretty far down the road.

That being the case, Duktape builds in two simple steps from the distributable or from the repo (without the Makefile):

The repo Makefile does these same steps. For example, 'make duk' will just run tools/configure.py and then invoke gcc once to compile all the sources in one step.

svaarala commented 7 years ago

One important point is that Duktape should place minimal requirements on whatever build system the application uses, be it Make, CMake, various IDEs, bash scripts etc -- the goal of the library is to be maximally embeddable to new and existing code bases without imposing limitations on what build systems can be used.

For that purpose the current model works reasonably well: the application writer incorporates a single Python execution (tools/configure.py) into their build system and compiles the resulting sources. Some users prefer to run configure.py manually and then just store the generated sources, which removes even that step from the build.

This is one reason Duktape distributable doesn't ship with supported build files: it would be difficult to maintain them for many targets, and it's actually awkward for application build systems if two different build systems are involved. Instead, the Python tooling essentially reduces the problem to "configuring" Duktape and then just straightforwardly compiling C files as you see fit.

There are no external library dependencies when Duktape itself is compiled. The main repo does have dependencies for tooling, running tests, etc, but as far as compiling duktape.c itself -- which is the main point -- the only dependency is libc. But application writers don't need to care about that: all they see is tools/configure.py and then compiling source files as is.

So this is roughly the background why I think the main use case, preparing duktape.c and compiling it, should be script based and the current model of there being a single script that the user runs (tools/configure.py) hides the internals well.

The top level Makefile is mostly just functionality around this basic core model:

This can be ported to CMake, but it only really affects people who want to modify Duktape itself and run test suites and such.

svaarala commented 7 years ago

Re: running tests indepdenently, you can say something like:

duktape@keppihevonen:~/duktape$ python util/runtest.py --duk ./duk tests/ecmascript/test-dev-mandel2-func.js 
SUCCESS test-dev-mandel2-func.js                           [ 0.0 s]

All that is needed is Python2.

rcdailey commented 7 years ago

Thanks for the details @svaarala. I just want to clarify that I'm not advocating CMake as a silver bullet. I think you give good examples of complex tasks that aren't really build-system related and can be handled in Python. In those examples, it sounds most appropriate to do a find_package(Python) to get access to Python within CMake (user must install Python prior to building your projects). After that, setting up a custom target (with add_custom_target()) to create a prerequisite build step that executes the Python scripts you have set up to perform each of those tasks.

The main thing I'm looking at right now is utils/configure.py, some of that can move to CMake because they're fundamental to setting up the build pipeline itself, for example the duktape.h.in configuration.

More detailed questions I have:

The general rule of thumb here for custom target is:

As long as we can decompose the various steps in your configure.py to follow this pre-condition/post-condition style of defining tasks, we can keep the logic in the python scripts and execute them intrinsically within the build process.

For a baby step right now, let's assume the following requirements (let me know if this is reasonable):

Based on this, what is the minimum required to get duktape.dll? I'll do my best to inspect the scripts as well but this will take longer and I'm doing this during my work hours. So any hours you can save me will benefit the amount of work I can do. Much appreciated!

svaarala commented 7 years ago

You should probably read these first, they outline the configuration and build process:

Re: user options, they're all command line options to configure.py, for example you give custom UnicodeData.txt using:

$ python tools/configure.py --unicode-data my_stripped_unidata.txt --special-casing my_stripped_specialcasing.txt

If configure.py were split into multiple parts and migrated into e.g. CMake, it would be a step down for people running different build systems and currently just running configure.py.

svaarala commented 7 years ago

The minimum for building a Windows DLL is:

$ python tools/configure.py --output-directory my-duktape --dll
# Then compile my-duktape/duktape.c as a DLL using 'cl'
rcdailey commented 7 years ago

I think if you plan to support CMake, perception will change:

So it's not really fair to call things a "step down", it's just different. If we need to have a discussion about the pros/cons or short/long term gains by switching to CMake, we can do that. Just note that CMake is designed to facilitate cross-platform builds. Python is not. You're doing a lot of hand-coding for solutions that CMake gives out of the box. There will be a little bit of back-tracking to get to feature parity, but I'm willing to do that work for you.

I just want to make sure we're on the same page about CMake vs configure.py before I start doing too much work. The expectation is that configuration.py goes away. That may entail moving some logic to CMake, while the rest of it goes into other Python scripts if needed.

rcdailey commented 7 years ago

To give you an overview, here is what the CMake build steps would look like:

  1. User runs cmake -G Ninja on linux to output 'ninja' build scripts (you can also use Make, there's a slew of different generators, which control which build system will be used to build the code). This is considered the "generation" or "configuration" step in CMake. All CMake scripts are executed, and targets build rules are generated into Ninja scripts (in this example).

    If the user needs to customize the build, they use -D to add more options, for example:

    cmake -G Ninja -D BUILD_SHARED_LIBS=ON

    This would cause CMake to generate shared library targets instead of static libraries. You can specify any number of options at this point. After generating, there is a CMakeCache.txt file in the output binary dir where you can make further configuration changes based on predefined cache variables and options.

  2. User runs ninja to execute the build based on the output ninja scripts in the previous step

    1. "ninja" would probably run the custom targets first (in your case) because we need some python scripts to run to output some files required for the build.

    2. Next targets to build are the C targets (e.g. libduktape)

  3. Run unit tests with ninja test

  4. User runs ninja install to install your library (headers, documentation, libraries, etc)

That's the gist of it.

svaarala commented 7 years ago

I just want to make sure we're on the same page about CMake vs configure.py before I start doing too much work. The expectation is that configuration.py goes away. That may entail moving some logic to CMake, while the rest of it goes into other Python scripts if needed.

Unfortunately I think we disagree on this goal.

I think this would be a downgrade for anyone not using CMake: they would still need Python2 because some of the preparation steps are complex enough to need scripting beyond a build file. But, instead of running a single Python command they would need to retrace the steps in the CMake files and replicate them in their own build system. Since those steps change from time to time, they would potentially need to redo this porting for Duktape releases.

Note that Duktape supports a very wide range of targets like Amiga, RiscOS, QNX, various embedded targets, even bare metal platforms (where Duktape runs inside the kernel and is built as part of the kernel). Right now things are extremely simple for the end user: use Python2 to run a single script and compile a single file, and this minimizes the assumptions made about the target, toolchain, or build system used.

I'm of course open to arguments why using CMake to replace configure.py would be an improvement over this. But such an argument needs to explain why things improve for people already having a non-CMake build system in place, and how non-mainstream targets would work.

I think so far the main question re: CMake has been to replace the repo Makefile with CMake. But as discussed above, this is more for Duktape developers, not application developers. I view the repo Makefile as rather inconsequential because application writers don't need it for anything. You don't even need if you just tweak sources in src-input/ and regenerate the Duktape files, as that happens with tools/configure.py too.

svaarala commented 7 years ago

Those using configure.py right now as the build system entry point would need to start using CMake instead

I think this is the main thing I disagree about (or maybe don't understand your argument):

I think the argument would make more sense if the dependency on Python2 was eliminated. But as far as it remains, I don't see why adding another dependency (CMake) makes things any easier. It may make things easier if you are already using CMake and prefer it, but for users that already use a different build system it would be an additional complication.

rcdailey commented 7 years ago

I think we're still on the wrong page here...

I think this would be a downgrade for anyone not using CMake: they would still need Python2 because some of the preparation steps are complex enough to need scripting beyond a build file

First, you only want one way of building your library. So users would have to use CMake. Second, we're not migrating all of the python code to CMake. You already made a very valid point as to the complexity of the work involved in those Python scripts. Instead, we embrace that by still requiring Python for the customization steps where it's necessary and more maintainable to do so. So when it's all said and done, the user is required to have both CMake and Python installed on their system. CMake will delegate to Python to run those Python scripts. Note that this means something significant from an architectural standpoint: CMake takes care of the build steps or build-specific tasks, and Python is now free to do non-build stuff, such as generating files, calculating stuff, etc. There is nothing wrong with this model. Users will not use Python directly. Instead, Python becomes an implementation detail managed by CMake, so users do not deal with it directly. Users use CMake, it takes care of the rest.

Note that Duktape supports a very wide range of targets like Amiga, RiscOS, QNX, various embedded targets, even bare metal platforms (where Duktape runs inside the kernel and is built as part of the kernel). Right now things are extremely simple for the end user: use Python2 to run a single script and compile a single file, and this minimizes the assumptions made about the target, toolchain, or build system used.

All of this is perfectly fine as long as those are targets. Host systems need to have access to CMake, but as long as you're just cross compiling you can easily support this in CMake via toolchain files. Known platforms and toolchains probably already have toolchain files out there you can use instead of having to make your own.

I'm of course open to arguments why using CMake to replace configure.py would be an improvement over this. But such an argument needs to explain why things improve for people already having a non-CMake build system in place, and how non-mainstream targets would work.

Part of what I was trying to say earlier is that benefits may not be measurable at this step alone. Obviously any benefits for replacing existing, working components are very small. There are more long term benefits, namely in supporting new build process and features going forward (You're using built-in CMake facilities instead of hand-coding this logic in Python scripts). The most important benefit for me at least, is that cross compiling becomes much easier. I have to cross compile duktape using the Android NDK toolchains. I cannot do this easily with your Python scripts. Fortunately for me, the NDK ships with a CMake toolchain file and I can use that to build your project using CMake with no extra effort beyond 1 additional command line argument.

I think so far the main question re: CMake has been to replace the repo Makefile with CMake. But as discussed above, this is more for Duktape developers, not application developers. I view the repo Makefile as rather inconsequential because application writers don't need it for anything. You don't even need if you just tweak sources in src-input/ and regenerate the Duktape files, as that happens with tools/configure.py too.

It's possible I could be misunderstanding, but to me it seems like you cannot simply make your project without running util/configure.py first. If that's true, then CMake does indeed combine those steps. Typically in linux you do:

./configure
make
make install

CMake simplifies this a little, because the build & configuration steps are not separated like that. The CMake scripts control what is both built and also configuration required to build.

svaarala commented 7 years ago

First, you only want one way of building your library.

I very much disagree here: I don't think Duktape should take any stance on what build system the user is using. As far as possible Duktape should just produce a set of C files to compile, in whatever way is most convenient for the build system the user is using. This is precisely why there are no official build files for Duktape.

This is mainly based on bad experience with supporting build files for user targets. There are just too many targets, too many build utility versions, etc, and sometimes people compile stuff on exotic platforms without cross-compilation tools. Even requiring Python2 has been a sticking point for some projects, and I'd gladly remove that dependency if there was an easy solution :)

I have to cross compile duktape using the Android NDK toolchains.

I would be quite surprised if you couldn't run tools/configure.py as a build pre-step.

rcdailey commented 7 years ago

Sorry I feel like we're talking over each other a little bit :-)

I'll try to respond to your last post and give some time to collect thoughts. I think we're misaligned a bit, because I understand CMake and you understand your build process, but we don't each understand both. I really apologize for not being able to properly explain myself.

One would still depend on Python2 because the things done by tools/configure.py are relatively complex such as autogenerating sources for ROM built-in objects. That is almost necessarily custom tooling.

You're right. At least initially, both CMake and Python will be required. CMake will execute python scripts for various tasks as needed. User won't see this happening, we will make CMake automate the Python parts.

Instead of depending on Python2 and running a single script that has reasonable versioning guarantees...

What does "reasonable versioning guarantees" mean?

...one is forced to adopt a specific build system and potentially integrate it into an already existing build system for the main application -- remember that Duktape is often a peripheral part of a much larger application.

If by "integrate" you mean: downstream projects use it? There's a few ways to do that easily with CMake.

  1. If the downstream project (i.e. the project that depends on duktape) is already using CMake, they would find your library on their system using find_package(duktape CONFIG). Doing this, CMake finds the includes and libraries on the host system for duktape. With this, they can build against your library.

  2. If they aren't using CMake, they are responsible for looking for the installed files of your library (i.e. they look in /usr/include, etc. for the files.

  3. If the are using CMake and want to build your library in real time, they can import your repository as a submodule in Git and then use CMake's add_subdirectory() to pull in your project and built it with theirs.

Targets that support Python2 but not CMake cannot be easily supported...

Not sure what this means...

...so one would need to retrace steps in CMake rather than just depend on Python2...

What does "retrace steps" mean? Are you talking about finding why a build failed?

...And then re-do this potentially for every Duktape release because the steps inside configure.py (which would now be inside CMake) change over time.

First of all, I'm not 100% certain all the contents of configure.py will be translated to CMake logic. It's possible that configure.py just gets split into multiple distinct python scripts, which CMake executes for the user.

I think the argument would make more sense if the dependency on Python2 was eliminated. But as far as it remains, I don't see why adding another dependency (CMake) makes things any easier. It may make things easier if you are already using CMake and prefer it, but for users that already use a different build system it would be an additional complication.

Eliminating the Python dependency is a long-term goal, not a short term one. One thing I've learned in our discussion and just poking around the code myself is that there is a lot of business logic in those python scripts. It's best to carry over the Python logic into CMake in small, manageable chunks where reasonable to do so. Also it's not even 100% clear to me at this point if all of the python logic should be implemented in CMake. Python has a huge framework full of functionality that CMake can't possibly compare with. Things that just make sense in Python should stay there.

I very much disagree here: I don't think Duktape should take any stance on what build system the user is using.

I just want to clarify terminology here, because your statement seems way off base:

CMake is not either of those. CMake doesn't build anything, instead it generates scripts for the build system, to use a specific build toolchain. This actually exceeds your requirement because CMake isn't forcing the user to use Make, or Visual Studio for example. It's letting the user pick what build system they want to use.

Secondly, and again just for clarity, I am not (and never have) intended to say that we should lock the user into any specific toolchain. That means, they can use clang, gcc, or whatever else.

You say that you take a stance on not requiring the user to use a build system, but you're doing exactly that: The only one true way to build your software is to use Python. This is no different from every other open source project out there: Some force users to use ./configure and make, others use Visual Studio. This is the main problem that CMake aims to solve: Specify the build rules, agnostic to the actual way that it's built. The user determines how they want to build the software, CMake just says what to build.

As far as possible Duktape should just produce a set of C files to compile, in whatever way is most convenient for the build system the user is using. This is precisely why there are no official build files for Duktape.

CMake gives you exactly this. This is why I can use the same CMake script to generate Visual Studio projects on Windows, then turn around and use them on Linux to generate Makefile scripts.

svaarala commented 7 years ago

@rcdailey Taking a complete different approach for a while: let's consider just the end user scenario for a while. Currently that is:

As discussed I'm rather hesitant to mess with this situation because it has very minimal dependencies on any aspect of the user's build system. (I'll also readily admit this is very much up to individual taste, and my taste has developer from working with a lot of tricky targets where standard tools don't always work.)

But, I would welcome the necessary CMake files to make this automatic for CMake users (without forcing everyone to use CMake) to make the life of CMake users easier.

The CMake parts would then just locate Python2, execute the configuration, and compile the resulting sources. The CMake parts would also need to translate between configure.py options and the CMake option model but that is relatively straightforward.

Would this be workable?

The repo Makefile is an entirely different issue IMO, because almost all of the targets are never used by ordinary users (and most not even by people floating a small patch set over Duktape mainline).

svaarala commented 7 years ago

Responses to some individual questions and clarifications:

What does "reasonable versioning guarantees" mean?

Running tools/configure.py with specific command line options will work without change for minor Duktape releases. However, the internals of configure.py are liable to change, but that's transparent to the end user. If configure.py is implemented as separated steps in CMake, then anyone not using CMake will need to check what the CMake files are doing and re-implement the steps in whatever they are using.

If they aren't using CMake, they are responsible for looking for the installed files of your library (i.e. they look in /usr/include, etc. for the files.

Duktape is not typically installed on the system as a library. It is most often compiled into an application as part of the application build. You can build it as a system library too, but that's not the main use case.

Not sure what this means...

Take any platform that supports Python2 but that is not supported by CMake. On that platform the users won't be able to use CMake and will need to reimplement whatever logic is inside CMake files.

What's more, some people compile Duktape on exotic targets with the target tools. They don't necessarily even have Python2, so they build the sources offline which works because the tooling doesn't need to understand anything about the targets as such, and then compile the sources on the target. Separating the preparation and compiling steps is exactly what is needed in this case.

Python has a huge framework full of functionality that CMake can't possibly compare with. Things that just make sense in Python should stay there.

For me, the whole process of preparing the sources is more naturally a Python-driven process. The process of compilation is build system specific, and because the interface between the two is a single Python command, I personally prefer to stay entirely out of the question of how the sources are ultimately compiled. There are just too many systems in use, and CMake doesn't support them all.

CMake is not either of those. CMake doesn't build anything, instead it generates scripts for the build system, to use a specific build toolchain. This actually exceeds your requirement because CMake isn't forcing the user to use Make, or Visual Studio for example. It's letting the user pick what build system they want to use.

What about platforms that don't run CMake? Or build systems or toolchains that CMake doesn't support -- say a custom IDE on a commercial Unix platform? Not everyone is using mainstream targets, and I'd much prefer to stay out of supporting their tools of choice :)

I just want to clarify terminology here, because your statement seems way off base:

What I mean by build system is everything dealing with how sources are prepared and compiled. I mean that just generically; if you use CMake to autogenerate Visual Studio files, these are both part of the "build system" as I view it. Maybe I should use a term like "build approach", and I really prefer staying out of that thing entirely, besides documenting how that single Python command is executed. After that it's a simple matter of compiling a single source file and the user can do that in whatever way is most convenient in their build approach.

You say that you take a stance on not requiring the user to use a build system, but you're doing exactly that: The only one true way to build your software is to use Python.

True in the sense that Python2 is needed to run the relatively complex preparation scripts. But you haven't proposed any change to this situation, but just adding a dependency to CMake in addition to the already existing Python2 dependency. CMake is not a solution to remove the Python dependency at all, so this argument is quite a red herring.

rcdailey commented 7 years ago

So what you're suggesting is:

  1. CMake automagically generates a command line invocation of configure.py based on a combination of target platform & user settings (which will be provided through cmake cache variables)
  2. CMake executes configure.py for the user
  3. CMake sets up targets to build the resulting source code

To me, if this is correct, it seems like a good baby step. However, there are certain things that configure.py is going that I'd rather CMake be doing. I'll give some specific examples:

  1. It generates duktape.h.in right now, I'd rather use CMake's configure_file() to fill in the @VARIABLE@ placeholders.

  2. It copies certain source files (that do not need intermediate processing) to a "temp directory" for compiling. This is something I'd rather do in CMake. Example code:

    From line 395:

    # Separate sources are mostly copied as is at present.
copy_files([
    'duk_alloc_default.c',
    'duk_api_internal.h',
    'duk_api_stack.c',
    ....snip....
    'duk_strings.h',
    'duk_replacements.c',
    'duk_replacements.h'
], srcdir, os.path.join(tempdir, 'src'))

    Reason I want CMake doing this, is because CMake manages the "tempdir" part for you automatically. There is a concept of "out of source" builds, i.e. basically CMake won't pollute your Git working directory with temporary files. It will generate all of the build system scripts and temp files (configure files) to a separate root directory that is completely disposable. However, since Python has no knowledge of this directory or the inner workings of CMake, it's best to let CMake do stuff related directly to the C files it will be building.

All of this is trivial to move over to CMake. The bigger issue here is that it will mean configure.py can't be used outside of CMake anymore. The easiest solution to this, as I was recommending earlier, is to only support 1 way of building your library. This means requiring users to use CMake directly instead of Python. It keeps things easier that way. I think the maintenance overhead of supporting 2 sets of build instructions outweighs the 1 time cost of installing CMake, which is trivial on most systems with package managers.

However, I'm assuming I'll lose that argument which is fine. If you want to keep both, I'll need to add some sort of switch to configure.py that turns off certain functionality. For example, cmake might invoke it with a --cmake-mode option that I can use to turn off certain tasks that CMake will do instead.

Thoughts?

rcdailey commented 7 years ago

Actually I'll do you one better: I'll make my first attempt at this do everything possible to not touch any python stuff. I'll see how far I can get.

I'm not able to make any arguments for one or the other, because I just don't have enough understanding of your build to make valid arguments. I apologize for the confusion. I thought this would be more straightforward but I'm apparently missing some key information.

I'll dig in, get my hands dirty, and give you a working example and we can go from there.

Thanks again so far for your support, this has been VERY helpful!

svaarala commented 7 years ago

It generates duktape.h.in right now, I'd rather use CMake's configure_file() to fill in the @VARIABLE@ placeholders.

Hmm... What purpose would this serve, besides a matter of style? Most of the replacements are rather mechanical for amalgamating files. There are maybe a few values, like the Duktape version, git commit information etc, that a user might want to control. Currently they can be given as command line options (--git-commit etc) for forcing specific values rather and autodetecting them.

It copies certain source files (that do not need intermediate processing) to a "temp directory" for compiling [...]

Instead of a --cmake-mode, would a simple --temp-dir option suffice? I.e. configure.py could be given an externally specified temp directory and it wouldn't delete it automatically afterwards?

In general it'd be nice to find a solution where configure.py didn't need to know it was running under CMake, with the addition of the necessary options to make that work.

rcdailey commented 7 years ago

Instead of a --cmake-mode, would a simple --temp-dir option suffice? I.e. configure.py could be given an externally specified temp directory and it wouldn't delete it automatically afterwards?

Does --output-directory not accomplish this? I'd probably place the output under ${CMAKE_BINARY_DIR}/configure_tmp. I'd just have to tell CMake where to expect the source files to build; I'm assuming they'd be under that directory.

svaarala commented 7 years ago

I'm not able to make any arguments for one or the other, because I just don't have enough understanding of your build to make valid arguments. I apologize for the confusion. I thought this would be more straightforward but I'm apparently missing some key information.

Some confusion stems from the fact that many people think that users agree on what "relevant" platforms are. For example, people working with Linux-OSX-Windows sometimes think e.g. Amiga or MS-DOS is an irrelevant platform and Duktape doesn't need to support it. While I appreciate that viewpoint, I do work with exotic targets and want to support them whenever it is reasonably possible, and I wouldn't want to break existing application builds for these targets.

As a concrete example, some users think that requiring C99 is OK because all "relevant" platforms support it; and if a target doesn't support it, you just inform the vendor to update it to C99 compliance. In practice this is quite difficult because vendors don't often have resources to do that for older platforms which may still be in use for years to come. And somehow MSVC still needs to be supported although it's not full C99, and I've never understood why people just don't email Microsoft to implement C99 because that's apparently easy to do ;)

I'll dig in, get my hands dirty, and give you a working example and we can go from there.

Sounds good!

svaarala commented 7 years ago

Does --output-directory not accomplish this? I'd probably place the output under ${CMAKE_BINARY_DIR}/configure_tmp. I'd just have to tell CMake where to expect the source files to build; I'm assuming they'd be under that directory.

--output-directory specifies the place where the final, ready-to-use files will appear: duktape.c, duktape.h, duk_config.h, and some JSON metadata in case the build needs to pick that up in later steps.

I could make configure.py use a temporary directory inside --output-directory and then delete it before finishing? But somehow it'd seem cleaner that the temp directory was separate and interrupted builds would never leave temporary files behind there.

rcdailey commented 7 years ago

As long as I have those final amalgamated source files, I can build & install duktape from CMake. This might be easier than we thought. I'm almost done with a sample that I can share with you.

svaarala commented 7 years ago

As long as I have those final amalgamated source files, I can build & install duktape from CMake. This might be easier than we thought. I'm almost done with a sample that I can share with you.

I think CMake would just need to run configure.py and then compile duktape.c. Assuming an amalgamated (single source duktape.c) configure run, the set of files to build is known in advance.

This should be quite straightforward, with the kinks being:

But, note that the original issue was about the repo Makefile which is quite a different thing because it does entirely different things unrelated to the end user :)

rcdailey commented 7 years ago

I assumed the Makefile was for building the generated source from configure.py. Is this incorrect? Sorry I'm all over the place on this :-(

I'm referring back to your 2nd post in this thread, where you have several bullet points. Are those all targets you need from Makefile to still work in CMake? Such as running the tests? Right now my immediate goal is getting it to build, but I can of course help with the other steps as I go.

svaarala commented 7 years ago

I assumed the Makefile was for building the generated source from configure.py. Is this incorrect? Sorry I'm all over the place on this :-(

The repo Makefile is never used to compile Duktape sources as part of a library or application build. It is only used by someone making Duktape changes and then wanting to compile various representative versions of Duktape for testing, benchmarking, etc. For example 'duk' is the default build, 'duk-low' is the default lowmem build, 'duk-low-rom' is a lowmem build with ROM built-in objects, and so on. End users should never need to use the top level Makefile and it isn't a part of the distributable (duktape-N.N.N.tar.xz).

In addition to the test executables the top level Makefile provides targets for API tests, Ecmascript tests, test262, Octane, regfuzz, etc, etc. It also provides targets to conveniently download some repos which are useful for testing (such as test262), and targets for making official source releases, website releases, etc. But this is only needed for Duktape development and is never part of the application build.

I'm referring back to your 2nd post in this thread, where you have several bullet points. Are those all targets you need from Makefile to still work in CMake? Such as running the tests? Right now my immediate goal is getting it to build, but I can of course help with the other steps as I go.

As far the scope of build needs related to the Duktape repo go:

I'm fine with prototyping the first part. It's the process that really matters, and must be as simple, well documented, and version stable as possible. The other two points are not really part of the supported, versioned release.

svaarala commented 7 years ago

As a concrete example of the distributable Makefiles, this repo+tag contains the 2.1.1 distributable unpacked into a git tag: https://github.com/svaarala/duktape-releases/tree/v2.1.1.

There are multiple example Makefiles whose only purpose is to illustrate what commands are needed, and Make is one way to communicate that. Application writers are not expected to use these Makefiles for anything related to their application build.

Concrete example, to build an eval tool: https://github.com/svaarala/duktape-releases/blob/v2.1.1/Makefile.eval

This particular example Makefile (and most if not all of the others) use the src/ directory of the distributable which contains pre-configured sources for the default configuration and platform autodetection. So in the simplest case, illustrated by the Makefiles, one doesn't need to run configure.py at all.

It would make sense to update the example Makefiles to be more informative though. They're pretty old and pre-date tools/configure.py by several years. But just to emphasize again, their only function is to illustrate how one could build Duktape as an example, not to recommend using these exact Makefiles. BAT files and bash scripts could be used for the same effect, if it conveyed the message across equally well.

fatcerberus commented 7 years ago

And somehow MSVC still needs to be supported although it's not full C99, and I've never understood why people just don't email Microsoft to implement C99 because that's apparently easy to do ;)

To be fair, it's getting there, finally. MSVC 2015-2017 have been good to me in this regard.

svaarala commented 7 years ago

So just to provide a very concrete example of how one builds one example application, the command line utility.

Preparing sources:

$ python tools/configure.py --output-directory /tmp/src
configure.py          INFO    Configuring Duktape version 2.1.99, commit 7f56ce4f88672de5dcebcbfb79c0e6eee60121e5, describe v2.1.0-559-g7f56ce4f, branch improve-duk-instanceof-no-prototype-error
configure.py          INFO      - source input directory: /home/duktape/duktape/src-input
configure.py          INFO      - license file: /home/duktape/duktape/LICENSE.txt
configure.py          INFO      - authors file: /home/duktape/duktape/AUTHORS.rst
configure.py          INFO      - config metadata directory: /home/duktape/duktape/config
configure.py          INFO      - output directory: /tmp/src
genconfig.py          INFO    Creating duk_config.h: platform=any, architecture=any, compiler=any
genconfig.py          WARNING Recommended config option DUK_USE_FATAL_HANDLER not provided
genbuiltins.py        INFO    Creating built-in initialization data: ram built-in support
genbuiltins.py        INFO    Removed 3 objects (0 disabled, 3 not needed by config), 8 properties (0 disabled, 8 not needed by config)
genbuiltins.py        INFO    Removed 3 objects (0 disabled, 3 not needed by config), 8 properties (0 disabled, 8 not needed by config)
combine_src.py        INFO    Combined 93 source files, 3506646 bytes written to /tmp/src/duktape.c
$ ls -l /tmp/src
total 3644
-rw-rw-r-- 1 duktape duktape   98548 Sep 11 23:25 duk_config.h
-rw-rw-r-- 1 duktape duktape   48370 Sep 11 23:25 duk_source_meta.json
-rw-rw-r-- 1 duktape duktape 3506646 Sep 11 23:25 duktape.c
-rw-rw-r-- 1 duktape duktape   69156 Sep 11 23:25 duktape.h

The compilation part depends on the application, but for the example command line tool:

$ gcc -o/tmp/duk -Os -std=c99 -I/tmp/src -Iexamples/cmdline /tmp/src/duktape.c examples/cmdline/duk_cmdline.c -lm
$ /tmp/duk
((o) Duktape 2.1.99 (v2.1.0-559-g7f56ce4f)
svaarala commented 7 years ago

To be fair, it's getting there, finally. MSVC 2015-2017 have been good to me in this regard.

I know, that doesn't really alter the argument though :)

So my main gripe is with the No True Scotsman style of argument where XXX is always right, and if it's not right, it's not really a relevant case. For people working on weird platforms, weird platforms are the norm and argumentation that they don't matter is not very convincing for them :)

fatcerberus commented 7 years ago

I didn't mean to imply not supporting MSVC was an option (it most definitely isn't :), I was just addressing your frustrations with MSVC's poor C99 support directly. Just throwing my two cents in, I don't really have a horse in this particular race. :)

svaarala commented 7 years ago

I actually don't have too much frustration with MSVC (maybe beside the C99 typing), a lot of older compilers are worse. If anything, I think MSVC is overwhelmingly worth supporting.

But I do have some frustration with "works for me" argumentation :)

svaarala commented 7 years ago

So basically I just wanted to illustrate why I would find it unconvincing for someone to say:

All proper platforms support C99, and thus MSVC doesn't need to be supported.

But this is precisely the argument I often hear to justify why target X doesn't need to be supported. It almost always comes from someone who doesn't need X themselves, and it's just a variation of the No True Scotsman argument.

For me there's no right or wrong policy for platform support. If one is happy with C99 and mainline platforms, that's fine. I'm not, and thus I also want to support more exotic platforms because I find them quite interesting. But you can't argue for or against it just be saying "X is not relevant". I also won't say I'm less or more "right" than someone advocating for C99-only support, it's just a premise rooted on what one is interested in.