Linux port

[nem0]

• [x] audio (missing effects) Effects will be reworked
• [x] debug
• [x] multiple windows no support for multiple windows on linux
• [x] text input
• [x] default cube fails to load
• [x] Continuous Integration
• [x] Command line
• [x] bugs

[MAnyKey]

At this point I stumbled upon include dirs. math_utils.cpp includes random: https://github.com/nem0/LumixEngine/blob/94d51b06070084d1e6b9bd76077b00eea7efcdfa/src/engine/core/math_utils.cpp#L4 which in turn includes <debug/debug.h> https://github.com/gcc-mirror/gcc/blob/master/libstdc%2B%2B-v3/include/std/random#L43 And with added include dir "src/engine" #include <debug/debug.h includes "src/engine/debug/debug.h" https://github.com/nem0/LumixEngine/blob/94d51b06070084d1e6b9bd76077b00eea7efcdfa/src/engine/debug/debug.h and everything is screwed. I don't know if genie has option to add include dir only for quote includes.

[nem0]

MSVS does not include <debug/debug.h> so it works on Windows. I guess we can rename debug.h to utils.h. If you have any better suggestions, I am listening.

[MAnyKey]

I thought about changing current include dirs. How about retain only "../src" and external includes? That way we hopefully add level of indirection and our own headers will not collide with system ones.

[nem0]

Sounds OK

[MAnyKey]

I found a good portion of Windows-specific code in core/mt. What do you think about replacing at least part of it with standard C++ threading code? As far as I'm concerned some of the current code contains UB (e.g. using volatile for the atomics, using union to type pun). Plus we can save the hassle of implementing std::mutex, std::lock_guard, std::condition_variable, std::atomic<> and spinlock on non-Win platforms.

[nem0]

After all my experiences with STL (although it was years ago) I really do not like to use it anywhere. I have tried to include and the preprocessed file went from 19kB to ~950kB. Also if included in every file, compile time went from ~25s to ~45s. That being said I have no problem with STL if it's encapsulated inside one cpp, i.e. if Linux uses STL's mutex internally.

As for undefined behaviour, there are places where it is intentional, e.g. I memcpy some classes with virtual functions somewhere. We need to discuss specific places. I would like to avoid using UB, but not if cost is too high, e.g. serious performance issues.

using volatile for the atomics, using union to type pun

I am not sure which code is this about.

[MAnyKey]

For unions: https://github.com/nem0/LumixEngine/blob/master/src/engine/core/mt/lock_free_fixed_queue.h#L52 https://github.com/nem0/LumixEngine/blob/master/src/engine/core/mt/lock_free_fixed_queue.h#L176 Dereferencing another part of union is safe in C11 but not in C++11.

for volatile: https://github.com/nem0/LumixEngine/blob/master/src/engine/core/mt/lock_free_fixed_queue.h#L195 https://github.com/nem0/LumixEngine/blob/master/src/engine/core/mt/sync.h#L92 I'm afraid there is no safe way to create atomic integers without C++11 or C11 atomics.

Also if included in every file, compile time went from ~25s to ~45s.

I envy you :smiley: , my experience with C++ compilation is much more sad.

Ok, I will try to isolate these changes from rest of the codebase.

[nem0]

Union: I think it was written by @tluqo this way for convenience. Feel free to change it if it causes any problems. volatile: I am not sure where exactly is the problem.

I've created a wiki page about undefined behaviour, you can put there anything you find. I will also go through all the code as I do not remember where I used UB and I bet there is myriad of places with UB I do not even know about.

my experience with C++ compilation is much more sad.

My experience is terrible. I tried to download one of the other smaller open source engines and compile it and it took like 5 minutes. And it even had less features than Lumix. That's why I put forward declarations or PIMPLs everywhere, and do all the other stuff that makes the code less readable. Yes, we can use precompiled header or unity builds, but they have their own problems and I want to avoid it as long as I can.

[MAnyKey]

It seems that crunch doesn't compile on x86_64 Linux. There is a pull request https://github.com/richgel999/crunch/pull/19 but it not merged since November. As an option we can fork it and reference our fork.

[nem0]

We can do that or switch from crunch to something else, I do not really care what library is used as long as it is simple, small and efficient. Do you know any better library we could use?

[nem0]

If you decide to use crunch, I've forked it https://github.com/nem0/crunch and add you as a collaborator.

[spacepluk]

Hi, I was giving this a try on Arch Linux and it compiles but when I run studio I get a SIGSEGV.

#0  0x00007ffff453b855 in __memmove_avx_unaligned_erms () from /usr/lib/libc.so.6
#1  0x00000000005631d6 in Lumix::DefaultAllocator::reallocate_aligned (this=0xc78380 <Lumix::PropertyRegister::getComponentTypes()::allocator>, ptr=0x0, size=224, align=4) at ../../../src/engine/default_allocator.cpp:67
#2  0x00000000005701a6 in Lumix::Array<Lumix::PropertyRegister::ComponentTypeData>::grow (this=0xc78390 <Lumix::PropertyRegister::getComponentTypes()::types>) at ../../../src/engine/array.h:312
#3  0x000000000056fde7 in Lumix::Array<Lumix::PropertyRegister::ComponentTypeData>::emplace<>() (this=0xc78390 <Lumix::PropertyRegister::getComponentTypes()::types>) at ../../../src/engine/array.h:221
#4  0x000000000056f841 in Lumix::PropertyRegister::getComponentType (id=0x90b30a "box_rigid_actor") at ../../../src/engine/property_register.cpp:143
#5  0x00000000004299ba in __static_initialization_and_destruction_0 (__initialize_p=1, __priority=65535) at ../../../src/physics/physics_scene.cpp:31
#6  0x0000000000429b16 in _GLOBAL__sub_I_physics_scene.cpp(void) () at ../../../src/physics/physics_scene.cpp:3708
#7  0x000000000090ad3d in __libc_csu_init ()
#8  0x00007ffff4433220 in __libc_start_main () from /usr/lib/libc.so.6
#9  0x000000000040943a in _start ()

[spacepluk]

Nevermind, I just noticed the binaries in external. Do you happen to have a script to configure/build that?

[spacepluk]

Hmm, I assumed that it was a mismatch in libc versions but it seems to be something else. Any idea what I might be doing wrong?

Thanks.

[nem0]

Nevermind, I just noticed the binaries in external. Do you happen to have a script to configure/build that?

https://github.com/nem0/lumixengine_3rdparty

Any idea what I might be doing wrong?

I've not run Linux build for months. You are not doing anything wrong, That code at https://github.com/nem0/LumixEngine/blob/master/src/engine/default_allocator.cpp#L67 is just wrong, it's copying size bytes while the old allocation can be smaller. I will have to think what to do with it and I am opened to any suggestions.

[spacepluk]

Thanks! I tried again and now I'm running into a different issue. This is what I'm doing:

• Cloned both LumixEngine and lumixengine_data
• Remove the asserts in platform_interface.cpp
• Compiled LumixEngine sh genie_linux_gcc.sh && make -j8 -C tmp/gcc/
• Replaced lumixengine_data/bin with a symlink to LumixEngine/project/tmp/gcc/bin/Debug
• Try to run ./bin/studio in lumixengine_data

Then I get a black windows and this. I'm hoping to get something similar to what you show in your video even if it's unstable. Is that unreasonable at the moment?

[nem0]

Hi, I am working on it, I will let you know as soon as it's fixed

[nem0]

@spacepluk I've commited some changes to both source code and data. Just make sure pipelines/shaderc is executable. Use forward rendering if you have a warning about unsupported blit.

You can ignore the assert in crunch library for now, it seems it causes no problems.

[spacepluk]

I forgot to mention that I had to copy gui/vera.ttg to bin/VeraMono.ttf. Now after doing that I think I'm just missing some files: https://gist.github.com/spacepluk/18ed0796e3e71a02ae45dd73e66e9594

[spacepluk]

hi there, I gave this another try today: https://gist.github.com/a79f2e2a142ea5eca915c26d31c36257

[nem0]

Hi, I have to implement mf_resource for linux and compile cmft library. I will try to fix this tonight or if it takes too much time then tomorrow.

[nem0]

It should compile now

[spacepluk]

thanks, I still can't get it to run on my system: https://gist.github.com/spacepluk/d3b6be3215ef23fd6d1bc6c88246f85e#file-studio-out-log

[nem0]

I don't have access to Linux machine now. I tried OpenGL on Windows and fixed found issues, I will try Linux on Monday probably.

[nem0]

Ok, I figured it out. All text files in pipelines/ directory (recursive) must be convertex from dos lineendings to unix lineendings

[dud3]

bash genie_linux_gcc.sh

Gives me:

genie_linux_gcc.sh: line 1: ./genie_linux: cannot execute binary file: Exec format error

[nem0]

@dud3 try to set genie_linux executable

[dud3]

@nem0 It seems like we also need sudo apt-get install g++-multilib before running make on tmp/gcc5, maybe it would be good to note them somewhere?

[dud3]

make inside tmp/gcc5 fails:

Linking app
/usr/bin/ld: cannot find -lGL
/usr/bin/ld: cannot find -lX11
collect2: error: ld returned 1 exit status
app.make:147: recipe for target 'bin/Debug/app' failed
make[1]: *** [bin/Debug/app] Error 1
Makefile:44: recipe for target 'app' failed
make: *** [app] Error 2

I've tried sudo apt-get install libgl1-mesa-dev for -lGL, but didn't work, also there are a lot of dependencies starting with libgl when searched on apt-get.

Any ideas?

[dud3]

Guessing that is 32bit issue, I'll try on 64bit some other day.

[nem0]

Added info about 64bit limitation on wiki for other people.