llebout / electron

:electron: Build cross-platform desktop apps with JavaScript, HTML, and CSS - PPC64LE FORK
https://electronjs.org
MIT License
6 stars 6 forks source link

Electron build fails on power... #1

Open junawaneshivani opened 4 years ago

junawaneshivani commented 4 years ago

Hi @leo-lb ,

I am trying to build electron on ppc64le Ubuntu by following the steps mentioned here... But it failed with below error.. Can you point me to the right script, or help me debug the issue here ? .. I would want to build the latest version of electron on power..

fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.
/root/Shivani/electron-gn/src/third_party/ffmpeg/ffbuild/common.mak:60: recipe for target 'libavcodec/ppc/vp8dsp_altivec.o' failed
make: *** [libavcodec/ppc/vp8dsp_altivec.o] Error 1
Traceback (most recent call last):
  File "./chromium/scripts/build_ffmpeg.py", line 927, in <module>
    sys.exit(main(sys.argv[1:]))
  File "./chromium/scripts/build_ffmpeg.py", line 524, in main
    options=options)
  File "./chromium/scripts/build_ffmpeg.py", line 903, in ConfigureAndBuild
    configure_args)
  File "./chromium/scripts/build_ffmpeg.py", line 897, in do_build_ffmpeg
    options.config_only, branding, configure_flags)
  File "./chromium/scripts/build_ffmpeg.py", line 430, in BuildFFmpeg
    ['make', '-j%d' % parallel_jobs] + libraries, cwd=config_dir)
  File "./chromium/scripts/build_ffmpeg.py", line 106, in PrintAndCheckCall
    subprocess.check_call(argv, *args, **kwargs)
  File "/usr/lib/python2.7/subprocess.py", line 541, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['make', '-j8', 'libavcodec/libavcodec.so.58', 'libavformat/libavformat.so.58', 'libavutil/libavutil.so.56']' returned non-zero exit status 2
llebout commented 4 years ago

@junawaneshivani Sorry I did not get notified

Use this container image as a base : registry.gitlab.com/chromium-ppc64le/ungoogled-chromium-ci-docker-image/ppc64le

Repo : https://gitlab.com/chromium-ppc64le/ungoogled-chromium-ci-docker-image

It's based on Fedora because there was an issue with Ubuntu, README isnt up to date

Also some dependencies will be missing I don't remember which, you'll see in the build logs and will be able to install them simply with dnf

llebout commented 4 years ago

And use 7-2-x branch it's the latest version I patched (if you need such a version)

llebout commented 4 years ago

Also what hardware are you on exactly?

llebout commented 4 years ago

Also look at prebuilds at : https://github.com/leo-lb/electron/releases

NishikantThorat commented 4 years ago

Hello Leo-lb,

I was trying to compile electron 7-2-x branch on UBUNTU (ppc64le), but not able to complete "gclient sync" command, as it failed due to missing gn binary.

Could you please share more details on issues on UBUNTU for electron build? We need to have electron on UBUNTU.

Any pointers will be helpful.

Thanks.

llebout commented 4 years ago

@NishikantThorat

Can you give exact logs?

Did you run https://raw.githubusercontent.com/leo-lb/electron/7-2-x/ppc64le_build.bash without cloning the repo beforehand? (You need to do this)

The script compiles and installs gn

NishikantThorat commented 4 years ago

Here is log snippet,

ib/npx.py", "yarn@1.15.2", "install", "--frozen-lockfile"]);'' took 23.13 secs ________ running '/usr/bin/python src/build/landmines.py' in '/root/nthorat/electron-gn' ________ running '/usr/bin/python src/third_party/depot_tools/update_depot_tools_toggle.py --disable' in '/root/nthorat/electron-gn' ________ running '/usr/bin/python src/tools/remove_stale_pyc_files.py src/android_webview/tools src/build/android src/gpu/gles2_conform_support src/infra src/ppapi src/printing src/third_party/blink/renderer/build/scripts src/third_party/blink/tools src/third_party/catapult src/third_party/closure_compiler/build src/tools' in '/root/nthorat/electron-gn' ________ running '/usr/bin/python src/buildtools/ensure_gn_version.py git_revision:152c5144ceed9592c20f0c8fd55769646077569b' in '/root/nthorat/electron-gn' Error: Command '/usr/bin/python src/buildtools/ensure_gn_version.py git_revision:152c5144ceed9592c20f0c8fd55769646077569b' returned non-zero exit status 1 in /root/nthorat/electron-gn /root/nthorat/electron-gn/src/buildtools/linux64/gn --versionfailed: Exec format error

NishikantThorat commented 4 years ago

Did you run https://raw.githubusercontent.com/leo-lb/electron/7-2-x/ppc64le_build.bash without cloning the repo beforehand? (You need to do this)

Yes, I copied the script and ran. No manual cloning.

llebout commented 4 years ago

@NishikantThorat Okay! Well the problem is that the gn in buildtools is first in the PATH, not sure why it's not done correctly in the script, because I built using it, but maybe my machine's PATH contained information that didnt trigger the error.

You need to modify the script and export PATH so that the built gn is first in the list!

llebout commented 4 years ago

It's stange because, if not PATH, that variable is set correctly, and depot_tools is frozen to my repo so it couldnt change in the mean time!

export DEPOT_TOOLS_GN="$(pwd)/gn/out/gn"
llebout commented 4 years ago

The other thing would be that you can fork the chromium repo and replace it within the DEPS file of this repo.

Then modify the DEPS file of Chromium:

Good luck!

llebout commented 4 years ago

Chromium repo is referenced here: https://github.com/leo-lb/electron/blob/7-2-x/DEPS#L81

llebout commented 4 years ago

Another thing that I see is that, nodejs version here:

https://github.com/leo-lb/electron/blob/7-2-x/DEPS#L16

is not the same as in the build script:

https://github.com/leo-lb/electron/blob/7-2-x/ppc64le_build.bash#L14

Please do make a pull request when you've got something!

llebout commented 4 years ago

The original issue with building Chromium (dependency of Electron) on Ubuntu is documented here:

https://gitlab.com/lle-bout/ungoogled-chromium/-/issues/6

NishikantThorat commented 4 years ago

It's stange because, if not PATH, that variable is set correctly, and depot_tools is frozen to my repo so it couldnt change in the mean time! export DEPOT_TOOLS_GN="$(pwd)/gn/out/gn"

Looks like, its picking incorrect gn binary,

# file gn_unittests
gn_unittests: ELF 64-bit LSB executable, _64-bit PowerPC_ or cisco 7500, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld64.so.2, for GNU/Linux 3.2.0, BuildID[sha1]=b8ffbb4649b3a96cb05911603f579f4e2bd020e0, stripped
# file gn
gn: ELF 64-bit LSB executable, _x86-64_, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, stripped
# ./gn_unittests
[496/496] XmlElementWriter.TestXmlEscape
_PASSED_
# pwd
/root/xx/gn/out
amitsadaphule commented 4 years ago

Hi @leo-lb, I was able to build "branch 7-2-x"/"tag v7.2.3-ppc64le" on Fedora 30 by making minor changes to the ppc64le_build.bash script (I'll raise a PR). I also needed to add this workaround to skip the gn version check.

But after a bit of digging, I found that you had removed ensure_gn_version dependency in this commit which is part of electron-ppc64le branch. But it's missing in electron-ppc64le-7-2-x. Could you please let me know if that commit was missed somehow or was it skipped on purpose with some other fix in place?

llebout commented 4 years ago

@amitsadaphule Great! It was most probably missed!

amitsadaphule commented 4 years ago

Thanks @leo-lb for the feedback! I'll raise a PR on https://github.com/leo-lb/chromium too.

amitsadaphule commented 4 years ago

@leo-lb I've raised the PR to chromium fork. Please have a look. Once that gets merged, I'll raise PR on this repo to update the chromium version in DEPS and another PR with minor change to the ppc64le build-script.

llebout commented 4 years ago

@amitsadaphule Hello! Here's the commit hash after the merge: 903f3e3c0059b21427b21139ae9ac55ed1223de2

You also need to specify that new commit hash here: https://github.com/leo-lb/electron/blob/4290618a13a6e612026aac8e3d8f42d5f16da663/DEPS#L14

amitsadaphule commented 4 years ago

Thanks @leo-lb! I've raised the PR on electron with updated chromium version.

llebout commented 4 years ago

@junawaneshivani @amitsadaphule @NishikantThorat

There is one more thing, it's that I adapt patches on occasion for new Electron versions, but this lacks process and maintenance work.

Would you be interested to help create and maintain forks of Electron in tandem with upstream supported releases? Along with @vddvss we maintain forks of Chromium and Ungoogled-Chromium with ppc64le patches here: https://gitlab.com/chromium-ppc64le - the difference with Electron is that it uses specific Chromium versions so patches for those versions would need to be maintained in parallel.

Just a reminder, we need to go through this process because Google is refusing our patches to Chromium. They show no interest in merging them, while they do have ppc64le test hardware in their infrastructure already and we offered to donate some to them if necessary. They already have a ppc64[le] port of V8 that IBM maintains upstream. Until proven otherwise, this is an act of bad faith on Google's side. We are not asking them to maintain the port in any way, just accept the patches, not offer any guarantee that it works, because it is troublesome to maintain them downstream, for various reasons that wouldnt exist if it was upstream.

So I am curious, what is your use case for Electron? Would you be willing to cooperate on maintenance for a ppc64le fork?

I am willing to lay out initial cleanup and work for this, then I could really use some help for day-to-day maintenance and update, as I some times have limited availability.

NishikantThorat commented 4 years ago

Hello @leo-lb,

On Ubuntu, I was trying to run unit tests after generating dist.zip, but renderer is getting crashed due to bad IPC message, after completing random percentage of tests (i.e. 15/ 35/ 82%). Could you please guide on investigating this issue?

tail error_msg.log

(node:1214) ProtocolDeprecateCallback: The callback argument of protocol module APIs is no longer needed. (node:1214) ProtocolDeprecateCallback: The callback argument of protocol module APIs is no longer needed. (node:1214) ProtocolDeprecateCallback: The callback argument of protocol module APIs is no longer needed. (node:1214) ProtocolDeprecateCallback: The callback argument of protocol module APIs is no longer needed. [1214:0806/043557.287168:INFO:CONSOLE(1120)] "document.registerElement is deprecated and will be removed in M80, around February 2020. Please use window.customElements.define instead. See https://www.chromestatus.com/features/4642138092470272 and https://developers.google.com/web/updates/2019/07/web-components-time-to-upgrade for more details.", source: /root/nthorat/electron-gn/src/electron/spec/chromium-spec.js (1120) [1214:0806/043557.604883:ERROR:validation_errors.cc(76)] Invalid message: VALIDATION_ERROR_UNEXPECTED_NULL_POINTER (null field 1) [1214:0806/043557.605046:ERROR:render_process_host_impl.cc(4582)] Terminating render process for bad Mojo message: Received bad user message: Validation failed for SpeechSynthesis RequestValidator [VALIDATION_ERROR_UNEXPECTED_NULL_POINTER (null field 1) [1214:0806/043557.605139:ERROR:bad_message.cc(27)] Terminating renderer for bad IPC message, reason 123 webContents 1 crashed: file:///root/nthorat/electron-gn/src/electron/spec/static/index.html?grep=&invert= (killed=true) Renderer process crashed



Is this the right way to execute unit test cases? How to ensure electron build is successful and completely functional as other platforms?

> Would you be willing to cooperate on maintenance for a ppc64le fork?

We are discussing the same and will get back to you soon.

Thanks.
gerrith3 commented 4 years ago

Hi @leo-lb I work with @NishikantThorat and we are willing to help to some extent with porting/supporting changes to enable ppc64-le. I am a bit worried about google's approach in not accepting community sponsored patches for a widely used package. I'm also a bit concerned about maintaining a full tree per version. Two things that would help us substantially here are (okay, three, with 0: being to get google to accept patches to top of tree ;-)), 1) minimizing the number of trees to support, ideally just one, possibly with branches for previous versions and 2) using a CI environment to build the top of tree, possibly on each commit. We do have ppc64le support in Travis CI now which is an ideal (free) option. It would be nice to also have a way to compare top of tree builds with the standard Intel builds from google to make sure we aren't chasing breakage in the ppc64le builds that also exists in the Intel build.

The problem is that our team is very small and the number of ppc64le packages that we build is rather large. Check out http://github.com/ppc64le/build-scripts for an idea of what we have currently ported.

So, we can commit to help, but the amount of help will be more of an "on request" or "background debugging" status, rather than some sort of full on co-maintainership, if that makes sense? Otherwise, do you have thoughts on what else we could help with to reduce the load on you, without having to pony up too much investment? ;) thanks!!

llebout commented 4 years ago

@gerrith3 To be clear, I am an individual (not company) doing voluntary work on this. I own a RaptorCS Talos II machine at home which I bought for a personal Free Software die-hard fantasy, a dream coming true for me. What I mean is shared maintenance, you would come and fix some problems and we share the work, it also helps me keep motivation. My needs for this Electron port are; run VS Codium, Signal Desktop, Wire and Riot.im on ppc64le. Most the initial work was completed by @shawnanastasio for Chromium on ppc64le and the rest is just fiddling with patch versions and dealing with Chromium build system. It's not expensive work, as in, it doesnt require deep ppc64le knowledge, you spend most time waiting for builds to finish.

The ideal situation would be that you depend on Electron as well especially and you share maintenance with me because you also need Electron for yourself.

In the eco-system, there's all sorts of users using different versions of Electron, supported releases are 7.x.x, 8.x.x, and 9.x.x currently - as written on: https://www.electronjs.org/docs/tutorial/support#currently-supported-versions

To ensure that the Electron port is useful, we need to support these three versions. It isnt necessarily lots of work to maintain, because I maintain ungoogled-chromium version-by-version since a while and therefore we have a patchset for every chromium version since a year and a half. If Electron doesnt use a Chromium version older than that we're good to go.

Feel free to try getting things merged with Google. Here's all the previous discussion: https://github.com/IBM/kui/issues/2201#issuecomment-519195663

llebout commented 4 years ago

Hello @leo-lb,

On Ubuntu, I was trying to run unit tests after generating dist.zip, but renderer is getting crashed due to bad IPC message, after completing random percentage of tests (i.e. 15/ 35/ 82%). Could you please guide on investigating this issue?

* Command,
#./out/Release/electron  electron/spec --enable-logging --no-sandbox
* Error logs,

# tail error_msg.log
(node:1214) ProtocolDeprecateCallback: The callback argument of protocol module APIs is no longer needed.
(node:1214) ProtocolDeprecateCallback: The callback argument of protocol module APIs is no longer needed.
(node:1214) ProtocolDeprecateCallback: The callback argument of protocol module APIs is no longer needed.
(node:1214) ProtocolDeprecateCallback: The callback argument of protocol module APIs is no longer needed.
[1214:0806/043557.287168:INFO:CONSOLE(1120)] "document.registerElement is deprecated and will be removed in M80, around February 2020. Please use window.customElements.define instead. See https://www.chromestatus.com/features/4642138092470272 and https://developers.google.com/web/updates/2019/07/web-components-time-to-upgrade for more details.", source: /root/nthorat/electron-gn/src/electron/spec/chromium-spec.js (1120)
[1214:0806/043557.604883:ERROR:validation_errors.cc(76)] Invalid message: VALIDATION_ERROR_UNEXPECTED_NULL_POINTER (null field 1)
[1214:0806/043557.605046:ERROR:render_process_host_impl.cc(4582)] Terminating render process for bad Mojo message: Received bad user message: Validation failed for SpeechSynthesis RequestValidator [VALIDATION_ERROR_UNEXPECTED_NULL_POINTER (null field 1)
[1214:0806/043557.605139:ERROR:bad_message.cc(27)] Terminating renderer for bad IPC message, reason 123
webContents 1 crashed: file:///root/nthorat/electron-gn/src/electron/spec/static/index.html?grep=&invert= (killed=true)
Renderer process crashed

Is this the right way to execute unit test cases? How to ensure electron build is successful and completely functional as other platforms?

Would you be willing to cooperate on maintenance for a ppc64le fork?

We are discussing the same and will get back to you soon.

Thanks.

Tests werent worked on, but I could run full-fledged apps successfully like Signal Desktop, VS Codium or Patchwork.

vddvss commented 4 years ago

@gerrith3 As you've seen, it's a beast to compile, and I have uncovered multiple compiler bugs compiling it.

It takes about 3-4 hours to build chromium using 16 threads and since I do LTO'd builds, about 32G of RAM. I think it would be about an half hour to an hour shorter without LTO, and would be somewhat quicker using a ramdisk.

At least for me, time and slow builds are the main constraints. We've been running CI on our own personal machines using gitlab, which has a local CI runner that can run on ppc64le.

I had looked at travis, but last I checked, the demands of compiling chromium are far beyond what the free tier of travis can handle--both in terms of time and system resources.

In terms of actually building ToT on every commit, it would be rather difficult because of the frequency of commits and resource constraints. Also, the google monorepo is rather difficult to work with (I mainly just work with a patchset in quilt). In the monorepo, the patches are against several trees:

The significant patches are against chromium and breakpad, with the other ones mainly being adding #ifdefs or fixing build scripts.

gerrith3 commented 4 years ago

@vddvss Agreed - that's a pretty big VM needed to compile electron - but we do have some ability to work with Travis on what sized VM and how long a project runs. I'll work with @NishikantThorat and @junawaneshivani to see if we can get an idea as to how much resource we would need at Travis and how long to build. We currently provide the back end servers for Travis and may be able to set up a larger/longer run for appropriately authorized projects. The primary goal, of course, is getting regression tests running automatically so no one has to kick them off and wait for their own server to plow through this build. There is a theme to some of these projects - taking a long time to build, needing more resources that the typical "free" offerings. But if you guys are willing, I'll see if we can find a path to make this all work with a public CI environment. I like that you are using a github runner currently but I assume you don't have the resources to build this on every change, as an example.

llebout commented 4 years ago

@gerrith3 Gitlab* runner, I could run on every change. The CI pipeline has to be optimized though, wont build everything from scratch on every commit. It needs caching.

I currently contribute Gitlab runners for various Free Software projects, currently more around GNOME and Flatpak because I am interested by desktop sandboxing but also for these chromium projects.

My home machine's configuration is:

Dual socket 8 core IBM POWER9 SMT4, 64 threads total 2x 2TB QVO Samsung SSDs available for runner VMs 256GB of DDR4 RAM

Does Electron use Travis-CI upstream? If you can donate resouces through Travis-CI that's awesome.