Is still Ubuntu 16.04 the suggested distro to build ?

[adriangibanelbtactic]

As I have already reported in https://github.com/ONLYOFFICE/build_tools/issues/802#issuecomment-2098265847 builds from Docker (which uses Ubuntu 16.04) seem to fail.

It probably arises from a depot_tools repo commit from April (one month ago) which forces you to use Python 3.6 or a newer version. I was wondering if, as it has already happened in the past OnlyOffice is built internally with some other dependencies and the Github repo is not updated yet.

I suspect that now internal builds are based on Ubuntu 20.04.

In that case both the README and the Dockerfile should be updated.

Steps to reproduce the problem:

cd /root
git clone \
  --depth=1 \
  --recursive \
  --branch v8.0.1.31 \
  https://github.com/ONLYOFFICE/build_tools.git \
  build_tools
# Ignore detached head warning
cd build_tools
mkdir out
docker build --tag onlyoffice-document-editors-builder .
docker run -e PRODUCT_VERSION=8.0.1 -e BUILD_NUMBER=31 -e NODE_ENV='production' -v $(pwd)/out:/build_tools/out onlyoffice-document-editors-builder /bin/bash -c 'cd tools/linux && python3 ./automate.py --branch=tags/v8.0.1.31'

build_tools base repo version: v8.0.1.31 Operating system: Ubuntu 16.04 Compiler version: Whatever Dockerfile installs.

[igwyd]

Hello @adriangibanelbtactic, you are right, i got the same error. I created ticket #68017

[shooding]

build_tools base repo version: v8.0.1.50 Operating system: Ubuntu 16.04

similar issue at building v8

return f'.{gcs_file_name}{MIGRATION_TOGGLE_FILE_SUFFIX}' ^ SyntaxError: invalid syntax

Due to need of python3.8

[igwyd]

It's fixed at https://github.com/ONLYOFFICE/build_tools/commit/c9de5278ea01fa45863ec0aa0bf1c7beba781dc4 and will be released in next major release. @adriangibanelbtactic and @shooding thank you for the detailed report.

[adriangibanelbtactic]

It's fixed at c9de527 and will be released in next major release. @adriangibanelbtactic and @shooding thank you for the detailed report.

Thank you very much @igwyd !

BTW, don't forget about the https://github.com/ONLYOFFICE/document-server-package/ repo. It has some issues that need some attention too.

Thank you again.

[shooding]

@igwyd release/v8.1.0 still gets error when building v8 and cipd

Cloning into '/core/Common/3dParty/v8_89/_gclient_v8_gdy3m7cc'...
remote: Sending approximately 789.65 MiB ...
remote: Counting objects: 7459, done
remote: Finding sources: 100% (3/3)
remote: Total 1098892 (delta 920986), reused 1098890 (delta 920986)
Receiving objects: 100% (1098892/1098892), 789.52 MiB | 9.11 MiB/s, done.
Resolving deltas: 100% (920986/920986), done.
Checking connectivity... done.

[0:03:50] Still working on:
[0:03:50]   v8
1>Syncing projects:   0% ( 0/ 2)
[0:03:52] Still working on:
[0:03:52]   v8
Syncing projects: 100% (38/38), done.

________ running 'python3 third_party/depot_tools/download_from_google_storage.py --no_resume --no_auth -u --bucket v8-wasm-spec-tests -s test/wasm-spec-tests/tests.tar.gz.sha1' in '/core/Common/3dParty/v8_89/v8'
  File "third_party/depot_tools/download_from_google_storage.py", line 51
    return f'.{gcs_file_name}{MIGRATION_TOGGLE_FILE_SUFFIX}'
                                                           ^
SyntaxError: invalid syntax
Error: Command 'python3 third_party/depot_tools/download_from_google_storage.py --no_resume --no_auth -u --bucket v8-wasm-spec-tests -s test/wasm-spec-tests/tests.tar.gz.sha1' returned non-zero exit status 1 in /core/Common/3dParty/v8_89/v8
  File "third_party/depot_tools/download_from_google_storage.py", line 51
    return f'.{gcs_file_name}{MIGRATION_TOGGLE_FILE_SUFFIX}'
                                                           ^
SyntaxError: invalid syntax

Subprocess failed with return code 2.
./cipd: line 137: ./depot_tools/cipd_client_version.digests: No such file or directory
Platform linux-amd64 is not supported by the CIPD client bootstrap: there's no pinned SHA256 hash for it in the *.digests file.
error: unknown option `type'

[adriangibanelbtactic]

It's fixed at c9de527 and will be released in next major release. @adriangibanelbtactic and @shooding thank you for the detailed report.

Wait a moment... that commit has a python3.tar.gz binary. That's not right to add binaries to a source repo. And, well, I can stand the many third party binaries you download because we can inspect which urls you are getting them from... but... Why we should trust a random binary?

I mean... that python3.tar.gz's python3/bin/python3.10 binary might have a trojan after all.

There should be another fix that does not end up throwing a binary on the repo.

Thank you for considering an alternative.

[igwyd]

@shooding, I cheked branch release/v8.1.0 4 days ago in docker 16.04 this error is gone, but branch release/v8.1.0 still in development and other error may appear.

@adriangibanelbtactic I think it's about trust to this repo, you are you are an experienced user and you can check binary files or replase it. I will show to developers issues from https://github.com/ONLYOFFICE/document-server-package/, thank you.

[shooding]

Struggling for several days

build for v7.5.1 [OK] serveral months ago. Host OS: Windows (x64) + make image from build_tools/Dockerfile (From Ubuntu:16.04)

build for v8.0.1 [FAILED] Host OS: macBook pro m1 (arm64) + make image from build_tools/Dockerfile (From Ubuntu:16.04) Host OS: Ubuntu 22.04 (x64) + make image from build_tools/Dockerfile (From Ubuntu:16.04) Host OS: Ubuntu 22.04 (x64) + make image from build_tools/Dockerfile (From Ubuntu:20.04)

Several common issues like:

1. "RPC failed; curl 18 transfer closed with outstanding read data remaining container"
2. missing header when building with clang++ (v19) but missing #include header, needs
3. remained build files from previous failure needs to clean up e.g. "core/Common/3dParty/boost/boost_1_72_0"

After all those fails l tried to update Dockerfile and automate.py "--update", "0" and mount pre-cloned repos like sdkjs and web-apps into the container to boost build speed in case downloading them many times.

In the updated Dockerfile pre-installed openjdk-11-jdk openjdk-11-jdk-headless openjdk-11-jre openjdk-11-jre-headless. Download qt_source_5.9.9.tar.xz, unzip and mount as qt-everywhere-opensource-src-5.9.9.

Even though tried configure.py with "--update", "0", still get error 1 when building core common modules like boost or v8. Should i just find a Ubuntu 16 machine as host and run python3 ./automate.py or find a Windows machine as host + docker build ?

[shooding]

I've managed to build for v8.0.1 [OK] Host OS: Ubuntu 22.04 (x64) + make image from build_tools/Dockerfile (From Ubuntu:16.04)

with several speed up techniques allow you to continue the build. Here's the Dockerfile that pre-install openjdk. You can omit this if network is very stable. Also change sources.list to the one closet helps.

FROM ubuntu:16.04

ENV TZ=Etc/UTC
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone

# Update the sources.list to use the mirror closet to you
RUN sed -i 's|http://archive.ubuntu.com/ubuntu/|http://free.nchc.org.tw/ubuntu/|g' /etc/apt/sources.list

RUN apt-get -y update && \
    apt-get -y install software-properties-common && \
    add-apt-repository ppa:openjdk-r/ppa && \
    apt-get -y update && \
    DEBIAN_FRONTEND=noninteractive apt-get -y install python \
                                    python3 \
                                    sudo \
                                    openjdk-11-jdk \
                                    openjdk-11-jdk-headless \
                                    openjdk-11-jre \
                                    openjdk-11-jre-headless \
                                    git \
                                    openssh-client

RUN rm /usr/bin/python && ln -s /usr/bin/python2 /usr/bin/python

VOLUME ["/work"]
WORKDIR /work

# Set the default command
CMD ["bash"]

export VERSION=8.0.1
cd build_tools && docker build -t documentserver-develop:$VERSION .

Note that i mounted ssh key into container so that we can force using ssh protocol to avoid "RPC failed; curl 18 transfer closed with outstanding read data remaining container" with unstable network (e.g. WiFi congestion). Note that in Dockerfile, using a volume /work to share the output with host machine, and you can edit/delete repos either within the container or host machine. This is helpful because sometimes you'll need to rm -rf core/Common/3dParty/boost/boost_1_72_0 to clean up intermediate build.

docker run -it --platform=linux/amd64 -p 80:80 \
-e PRODUCT_VERSION=$VERSION \
-e BUILD_NUMBER=50 \
-v ~/.ssh:/root/.ssh \
-v $WORKFOLDER:/work \
documentserver-develop:$VERSION

You'll enter the container then copy & paste these command to have smooth cloning process.

git config --global url."git@github.com:".insteadOf "https://github.com/" && ssh-keyscan github.com >> /root/.ssh/known_hosts && eval "$(ssh-agent -s)" && ssh-add /root/.ssh/id_rsa

If you github ssh key is encrypted, here you enter password. In my senario, i just want to change some stuff in sdjks and web-apps, therefore building server is enough, but still want consistency so pre-cloned these repos and checkout to the correct tag e.g. 8.0.1.50 accordinly, then mount to /work volume before building.

# To share with host machine. Mount -v /path/from/host:/work
# ├── build_tools
# ├── core
# ├── core-fonts
# ├── dictionaries
# ├── qt-everywhere-opensource-src-5.9.9 (pre download and tar -xf )
# ├── sdkjs
# ├── server
# ├── web-apps
# ├── document-server-integration
# ├── ...

You can check the function def get_repositories() in scripts/base.py and comment out those repos pre-cloned. Like:

def get_repositories():
  result = {}
  # result["core"] = [False, False]
  # result["sdkjs"] = [False, False]
  result.update(get_sdkjs_addons())
  result["onlyoffice.github.io"] = [False, False]
  # result["web-apps"] = [False, False]
  result.update(get_web_apps_addons())
  # result["dictionaries"] = [False, False]
  # result["core-fonts"] = [False, False]

Why need to comment out manually? while def update_repositories(repositories) should've done the git update ? This is because core repo is a submodule of DocumentServer and the building script will check it's parent and error: fatal: Not a git repository: ../.git/modules/core

Then i ran with

cd build_tools/tools/linux/ && python3 ./automate.py server

Build [OK]