search

hoehermann / purple-gowhatsapp

Pidgin/libpurple plug-in for WhatsApp Web.
GNU General Public License v3.0
290 stars 34 forks source link

Compilation speed #175

Open beadon opened 1 year ago

beadon commented 1 year ago

With the recent issue related to building, I was reminded of the time it took you to download and compile the required sources.

For kicks I wanted to see how far apart we were for compilation performance.

Timing the local compilation I am downloading and building in less than 7 seconds. You had mentioned it was taking "minutes" to do the same from a clean repo oull. What could possibly be taking so long on your side ?

reference 'dumb' compilation test:

beadon@semiauto:~/build/purple-gowhatsapp$ time ( mkdir build && cd build && cmake .. && cmake --build . )
-- The C compiler identification is GNU 12.2.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Checking for module 'purple'
--   Found purple, version 2.14.12
-- PURPLE_INCLUDE_DIRS: /usr/include/libpurple;/usr/include/glib-2.0;/usr/lib/x86_64-linux-gnu/glib-2.0/include
-- PURPLE_LIBRARY_DIRS: /usr/lib/x86_64-linux-gnu
-- PURPLE_LIBRARIES: purple;glib-2.0
-- PURPLE_PLUGIN_DIR: /usr/lib/x86_64-linux-gnu/purple-2
-- PURPLE_DATA_DIR: /usr/share
-- Checking for module 'gdk-pixbuf-2.0'
--   Found gdk-pixbuf-2.0, version 2.42.10
-- Checking for module 'opusfile'
--   Found opusfile, version 0.12
-- PIXBUF_INCLUDE_DIRS: /usr/include/gdk-pixbuf-2.0;/usr/include/glib-2.0;/usr/lib/x86_64-linux-gnu/glib-2.0/include;/usr/include/libpng16;/usr/include/x86_64-linux-gnu;/usr/include/libmount;/usr/include/blkid
-- OPUSFILE_INCLUDE_DIRS: /usr/include/opus
-- OPUSFILE_LIBRARIES: opusfile
-- Configuring done
-- Generating done
-- Build files have been written to: /home/beadon/build/purple-gowhatsapp/build
[  4%] Building C object src/go/CMakeFiles/purple-dummy.dir/purple-dummy.c.o
[  8%] Building C object src/go/CMakeFiles/purple-dummy.dir/__/c/constants.c.o
[ 12%] Linking C static library libpurple-dummy.a
[ 12%] Built target purple-dummy
[ 16%] Generating purple-go-whatsapp.a, purple-go-whatsapp.h
go: downloading github.com/alfg/mp4 v0.0.0-20210728035756-55ea58c08aeb
go: downloading github.com/go-sql-driver/mysql v1.6.0
go: downloading github.com/lib/pq v1.10.4
go: downloading github.com/mattn/go-sqlite3 v1.14.10
go: downloading github.com/mdp/qrterminal/v3 v3.0.0
go: downloading github.com/skip2/go-qrcode v0.0.0-20200617195104-da1b6568686e
go: downloading go.mau.fi/whatsmeow v0.0.0-20230727120640-c313a80ab292
go: downloading golang.org/x/net v0.9.0
go: downloading google.golang.org/protobuf v1.30.0
go: downloading rsc.io/qr v0.2.0
go: downloading go.mau.fi/libsignal v0.1.0
go: downloading golang.org/x/crypto v0.8.0
go: downloading filippo.io/edwards25519 v1.0.0
go: downloading github.com/gorilla/websocket v1.5.0
go: downloading golang.org/x/text v0.9.0
[ 20%] Built target purple-go-whatsapp
[ 25%] Building C object src/c/CMakeFiles/whatsmeow.dir/init.c.o
[ 29%] Building C object src/c/CMakeFiles/whatsmeow.dir/login.c.o
[ 33%] Building C object src/c/CMakeFiles/whatsmeow.dir/qrcode.c.o
[ 37%] Building C object src/c/CMakeFiles/whatsmeow.dir/bridge.c.o
[ 41%] Building C object src/c/CMakeFiles/whatsmeow.dir/process_message.c.o
[ 45%] Building C object src/c/CMakeFiles/whatsmeow.dir/display_message.c.o
[ 50%] Building C object src/c/CMakeFiles/whatsmeow.dir/groups.c.o
[ 54%] Building C object src/c/CMakeFiles/whatsmeow.dir/blist.c.o
[ 58%] Building C object src/c/CMakeFiles/whatsmeow.dir/constants.c.o
[ 62%] Building C object src/c/CMakeFiles/whatsmeow.dir/send_message.c.o
[ 66%] Building C object src/c/CMakeFiles/whatsmeow.dir/handle_attachment.c.o
[ 70%] Building C object src/c/CMakeFiles/whatsmeow.dir/send_file.c.o
[ 75%] Building C object src/c/CMakeFiles/whatsmeow.dir/presence.c.o
[ 79%] Building C object src/c/CMakeFiles/whatsmeow.dir/options.c.o
[ 83%] Building C object src/c/CMakeFiles/whatsmeow.dir/receipt.c.o
[ 87%] Building C object src/c/CMakeFiles/whatsmeow.dir/pixbuf.c.o
[ 91%] Building C object src/c/CMakeFiles/whatsmeow.dir/commands.c.o
[ 95%] Building C object src/c/CMakeFiles/whatsmeow.dir/opusreader.c.o
[100%] Linking C shared library libwhatsmeow.so
[100%] Built target whatsmeow

real    0m6.894s
user    0m4.383s
sys 0m1.149s
beadon commented 1 year ago

Just for kicks, even faster ...

beadon@semiauto:~/build/purple-gowhatsapp$ time ( mkdir build && cd build && cmake .. && cmake --build . -j4 )
-- The C compiler identification is GNU 12.2.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Checking for module 'purple'
--   Found purple, version 2.14.12
-- PURPLE_INCLUDE_DIRS: /usr/include/libpurple;/usr/include/glib-2.0;/usr/lib/x86_64-linux-gnu/glib-2.0/include
-- PURPLE_LIBRARY_DIRS: /usr/lib/x86_64-linux-gnu
-- PURPLE_LIBRARIES: purple;glib-2.0
-- PURPLE_PLUGIN_DIR: /usr/lib/x86_64-linux-gnu/purple-2
-- PURPLE_DATA_DIR: /usr/share
-- Checking for module 'gdk-pixbuf-2.0'
--   Found gdk-pixbuf-2.0, version 2.42.10
-- Checking for module 'opusfile'
--   Found opusfile, version 0.12
-- PIXBUF_INCLUDE_DIRS: /usr/include/gdk-pixbuf-2.0;/usr/include/glib-2.0;/usr/lib/x86_64-linux-gnu/glib-2.0/include;/usr/include/libpng16;/usr/include/x86_64-linux-gnu;/usr/include/libmount;/usr/include/blkid
-- OPUSFILE_INCLUDE_DIRS: /usr/include/opus
-- OPUSFILE_LIBRARIES: opusfile
-- Configuring done
-- Generating done
-- Build files have been written to: /home/beadon/build/purple-gowhatsapp/build
[  8%] Building C object src/go/CMakeFiles/purple-dummy.dir/purple-dummy.c.o
[  8%] Building C object src/go/CMakeFiles/purple-dummy.dir/__/c/constants.c.o
[ 12%] Linking C static library libpurple-dummy.a
[ 12%] Built target purple-dummy
[ 16%] Generating purple-go-whatsapp.a, purple-go-whatsapp.h
go: downloading github.com/alfg/mp4 v0.0.0-20210728035756-55ea58c08aeb
go: downloading github.com/go-sql-driver/mysql v1.6.0
go: downloading github.com/lib/pq v1.10.4
go: downloading github.com/mattn/go-sqlite3 v1.14.10
go: downloading github.com/mdp/qrterminal/v3 v3.0.0
go: downloading github.com/skip2/go-qrcode v0.0.0-20200617195104-da1b6568686e
go: downloading go.mau.fi/whatsmeow v0.0.0-20230727120640-c313a80ab292
go: downloading golang.org/x/net v0.9.0
go: downloading google.golang.org/protobuf v1.30.0
go: downloading rsc.io/qr v0.2.0
go: downloading go.mau.fi/libsignal v0.1.0
go: downloading golang.org/x/crypto v0.8.0
go: downloading filippo.io/edwards25519 v1.0.0
go: downloading github.com/gorilla/websocket v1.5.0
go: downloading golang.org/x/text v0.9.0
[ 20%] Built target purple-go-whatsapp
[ 37%] Building C object src/c/CMakeFiles/whatsmeow.dir/init.c.o
[ 37%] Building C object src/c/CMakeFiles/whatsmeow.dir/login.c.o
[ 37%] Building C object src/c/CMakeFiles/whatsmeow.dir/qrcode.c.o
[ 37%] Building C object src/c/CMakeFiles/whatsmeow.dir/bridge.c.o
[ 54%] Building C object src/c/CMakeFiles/whatsmeow.dir/groups.c.o
[ 54%] Building C object src/c/CMakeFiles/whatsmeow.dir/blist.c.o
[ 54%] Building C object src/c/CMakeFiles/whatsmeow.dir/process_message.c.o
[ 54%] Building C object src/c/CMakeFiles/whatsmeow.dir/display_message.c.o
[ 58%] Building C object src/c/CMakeFiles/whatsmeow.dir/constants.c.o
[ 62%] Building C object src/c/CMakeFiles/whatsmeow.dir/send_message.c.o
[ 66%] Building C object src/c/CMakeFiles/whatsmeow.dir/handle_attachment.c.o
[ 70%] Building C object src/c/CMakeFiles/whatsmeow.dir/send_file.c.o
[ 75%] Building C object src/c/CMakeFiles/whatsmeow.dir/presence.c.o
[ 79%] Building C object src/c/CMakeFiles/whatsmeow.dir/options.c.o
[ 83%] Building C object src/c/CMakeFiles/whatsmeow.dir/receipt.c.o
[ 87%] Building C object src/c/CMakeFiles/whatsmeow.dir/pixbuf.c.o
[ 91%] Building C object src/c/CMakeFiles/whatsmeow.dir/commands.c.o
[ 95%] Building C object src/c/CMakeFiles/whatsmeow.dir/opusreader.c.o
[100%] Linking C shared library libwhatsmeow.so
[100%] Built target whatsmeow

real    0m3.669s
user    0m3.922s
sys 0m1.114s

sub 4 seconds ...

hoehermann commented 1 year ago

A couple of factors.

This is especially pronounced for the win32 build:

$ time cmake --build .
Consolidate compiler generated dependencies of target purple-dummy
[ 12%] Built target purple-dummy
[ 16%] Generating purple-go-whatsapp.a, purple-go-whatsapp.h
go: downloading github.com/alfg/mp4 v0.0.0-20210728035756-55ea58c08aeb
go: downloading github.com/go-sql-driver/mysql v1.6.0
go: downloading github.com/lib/pq v1.10.4
go: downloading github.com/mattn/go-sqlite3 v1.14.10
go: downloading github.com/mdp/qrterminal/v3 v3.0.0
go: downloading github.com/skip2/go-qrcode v0.0.0-20200617195104-da1b6568686e
go: downloading go.mau.fi/whatsmeow v0.0.0-20230727120640-c313a80ab292
go: downloading golang.org/x/net v0.9.0
go: downloading google.golang.org/protobuf v1.30.0
go: downloading rsc.io/qr v0.2.0
go: downloading go.mau.fi/libsignal v0.1.0
go: downloading golang.org/x/crypto v0.8.0
go: downloading filippo.io/edwards25519 v1.0.0
go: downloading github.com/gorilla/websocket v1.5.0
go: downloading golang.org/x/text v0.9.0
[ 20%] Built target purple-go-whatsapp
Consolidate compiler generated dependencies of target whatsmeow
[ 25%] Building C object src/c/CMakeFiles/whatsmeow.dir/init.c.obj
[ 29%] Building C object src/c/CMakeFiles/whatsmeow.dir/login.c.obj
[ 33%] Building C object src/c/CMakeFiles/whatsmeow.dir/process_message.c.obj
[ 37%] Building C object src/c/CMakeFiles/whatsmeow.dir/groups.c.obj
[ 41%] Building C object src/c/CMakeFiles/whatsmeow.dir/blist.c.obj
[ 45%] Building C object src/c/CMakeFiles/whatsmeow.dir/send_message.c.obj
[ 50%] Building C object src/c/CMakeFiles/whatsmeow.dir/send_file.c.obj
[ 54%] Building C object src/c/CMakeFiles/whatsmeow.dir/presence.c.obj
[ 58%] Building C object src/c/CMakeFiles/whatsmeow.dir/receipt.c.obj
[ 62%] Building C object src/c/CMakeFiles/whatsmeow.dir/commands.c.obj
[ 66%] Linking C shared library libwhatsmeow.dll
[100%] Built target whatsmeow

real    1m41,724s
user    0m0,012s
sys 0m0,046s
beadon commented 1 year ago

/I suspect the CPU, as you do, but I suspect your storage is the real slowdown. Given the age of the hardware and the ready availability of components it's cheap to update the CPU and the storage relatively easily.

Here's that fastest processor for that socket -- should fit, give a nice performance boost, about $29 USD (free shipping): https://www.ebay.com/sch/i.html?_from=R40&_trksid=p4432023.m570.l1313&_nkw=i7+3770&_sacat=0

Next up the storage - likely SATA, your primary drive likely is short on the available IOPS. About $58 USD for this 1TB , very speedy, SSD: https://www.ebay.com/itm/354850276245

If you want to really go fast, then a M.2 to PCIE adaptor will let you step up to the next level , ~$6 : https://www.ebay.com/itm/292931456826

And then select a M.2 drive, which can be had for $35 -- prices are recently falling dramatically.

Networking -- 1.4 MB/s works out to about 11 mbps ( very close to the 10mbps of a physical link problem), so, if you sure this is your ISP speed is correct don't worry. But, your downloads there are quite small, I will need to see how much space is truly required, this should still be only a few seconds. If you are not getting this speed from the ISP, and actually have something higher, then it's time to look at ethernet cables :)

I hope these will help. For a few dollars your performance will be greatly improved. I anticipate you may even be able to find better deals locally.

hoehermann commented 1 year ago

Thank you for the suggestions.

Those prices reflect on the US market. In Europe, prices did not really go down. I blame the pandemic and the Russo-Ukraine war. With an international purchase, I would need to pay import tax. By the way, the CPU you recommended would only give me a slight boost, according to userbenchmark.com. The time I would spend on the upgrade process would probably exceed the benefits. Also, more and more USB-ports on my system's main-board are failing. I will probably update some day anyways. ;)

However, not everyone can do. While working on this project for the past years, I communicated with users from Africa and South America. They are stuck with 32bit systems, sometimes of the Pentium 4 era. They depend on projects like this since they cannot use the official WhatsApp Destkop client due to the limitations of their systems. I see no reason to make their life even harder by forcing a cache wipe into the build process. That was – if I remember correctly – the context I mentioned the long build times in. :)

EionRobb commented 1 year ago

Might go even faster if you don't build to the intermediate obj/o files and go straight to dll/so by compiling all the c files at once (can get the benefit of not having to scan through all the header includes for multiple times, for each c file).

Speeds up compile times by about x2 for some of the other pidgin plugins I worked on, hopefully gives a boost here too?

hoehermann commented 1 year ago

On my system, recompiling the plug-in's go part takes 6 seconds on my system. Recompiling the entire C part takes 2 seconds. I have no intention to speed that up.

In a previous contribution #166, Bryant opted to drop the build cache prior to building. This will make go re-download and re-compile all dependencies. That I deem unacceptable.

On a side-note: I just noticed that when using CGO (go's ffi features which we have to use), the go build process is single-threaded. According to this thread, it seems to be by design. More cores will not help in this matter.

beadon commented 1 year ago

The goal is within this thread only : Improve a developer's workflow, based on the metric of compilation speed. Specifically yours!

I am not pulling any other threads in here.

It's good to know about the 32-bit users in other countries. I would recommend then that we keep some kind of 32-bit compilation around to support these users.

I was identifying key parts of your workflow that would speed up this compilation, and probably really improve anything else you do with the machine. I was citing costs because I wasn't necessarily expecting you to buy things, but perhaps the community could ? (these are relatively modest parts, some people might even have them around)

Thoughts ?

beadon commented 1 year ago

A quick note on multi-threading the compile, notice the speedup I applied in the second try "cmake --buld . -j4 " not because I thought it should be applied, but just because that was proof the ~7s could still be improved if necessary (and as expected with the right parallelization).

That being said, on rereading your last comment -- believe I am reading that you are quoting the go downloading parts to your machine at approximately 1min 29s? ( other parts of the build about 7s total ? ).