Switch to CMake and git submodules

joeyparrish commented 2 years ago

The use of our ancient chromium build infrastructure has been increasingly problematic. We use old versions of chromium components so that they are compatible with gyp (gyp has been removed upstream), but this means also an older version of depot_tools, and lack of support for newer OSes, SDKs, and Linux distros.

We should replace gyp with CMake, gclient with submodules, and depot_tools should go away completely. Dependencies (like protobuf, etc) should be updated to the latest versions, and if possible, system versions should be used instead of building from source.

I'm not sure what order these should be done in, or how much work they will be. I'll experiment with it a bit to figure that out, but I may or may not be the one to finish the work. Volunteers are welcome!

joeyparrish commented 2 years ago

Oh, and of course, some of the old Python build scripts in our old forks of Chromium repos will only run with Python 2, which is not even installed by default any more in many places.

joeyparrish commented 2 years ago

In a pure python3-environment, such as my workstation at the office, I can't even run gyp any more. So replacing gyp is a high priority in this list.

Replacing gclient with submodules is relatively easy and straightforward. See https://github.com/joeyparrish/shaka-packager/commit/f4d560ed924c54c7ec97389d19bd5b5d25f02b9e Some of those deps could be dropped by the time this work is complete (such as gyp, packager/build).

zdanek commented 2 years ago

@joeyparrish I managed to fix some problems with generating CMakeLists.txt from gyp. It was discussed some time ago at https://github.com/shaka-project/shaka-packager/pull/774 I wished to make a PR to fix issues with generation of CMakeLists.txt but I failed when discovered that some tools are part of Chromium ecosystem and is very hard to fix because those tools were from ancient revision of Chromium tools.

But still - I can tell how to fix locally some files in order to generate CMakeLists.txt.

And by the way - I use Python3 with Shaka gyp and it works (AFAIK I had to fix only about two places regarding py2->py3)

joeyparrish commented 2 years ago

@zdanek, thank you! You shouldn't need to make permanent changes to gyp. You could make temporary changes, generate CMakeLists.txt from gyp, then clean up CMakeLists.txt by hand and commit it. The goal is to get rid of gyp, so it should be fine to hack it up as needed if it is easier than writing CMakeLists.txt completely from scratch.

zdanek commented 2 years ago

@joeyparrish I got it.

I was using generated CMakeLists.txt for CLion to edit files and have includes working properly. It was OK but I never compiled Shaka with make. I always used command line and ninja like everybody. Now I tried to builld it and it failed.

There are a few problems:

it's very very messy
it doesn't compile
release is quite different than debug - but we need both (merged into one CMakeLists.txt)
it's flat - I mean, that every directory / library / third party lib has it's own section put into a single CMakeLists.txt - it should be divided into smaller files, probably for directories / modules

Biggest problem is that it does not compile. That means, some paths are messed up and some includes are bad defined so inside the source files includes doesn't work. If there's no other volunteer I might spend some evenings trying to clean it up and/or rethink better approach. Maybe it will need more people to help to split it into sub-works to speed it up and not to be so boring and arduous task :)

I will try to dig into it tomorrow. First task is to build Shaka using cmake.

joeyparrish commented 2 years ago

I see. Thank you for the update!

It seems there isn't a better solution right now than manually creating the build files for cmake. I agree that doing it one library/dependency at a time is probably best.

joeyparrish commented 2 years ago

I have begun the process of writing CMake files by hand for each library.

joeyparrish commented 2 years ago

As part of this work, ~C++14~ C++17 will now be required to build Shaka Packager. This will reduce our dependencies, and we will rely more on the standard library.

zdanek commented 2 years ago

@joeyparrish regarding to write a new CMake files by somebody from the community - I was thinking about it as I wrote 3 months ago but as you currently see it's quite absorbing and in my opinion must be done by somebody who is backed by bigger company that pays for the time spent. I wouldn't have enough time to do it. The time that I need to spend on my current job. The transition is hard. I'm glad that you decided to move it forward. Thank you on behalf of all developers depending on Shaka Project :)

joeyparrish commented 2 years ago

Well, I only get to spend one day a week on it, so it will take a long time. Thankfully, @kqyang has offered to do code reviews as I make progress.

I'll start merging partial build system work to a branch in stages. If anyone is interested, they can follow along here. I'll tag this issue with each PR, and you can play with the branch I'm creating and provide feedback.

So far, I've got 4 internal libraries building, and I've ripped out Chromium build tools, Chromium base, and gflags. I've updated curl, boringssl, gmock, and gtest. I've added abseil and glog.

Those 4 internal libraries build, including their tests, but I'm stuck on crashing tests in the file library.

zdanek commented 2 years ago

Although local problems (crashing tests) this is good news. Getting rid of Chromium build tools is my dream since 1.5 years :) Great decisions. I will take a look at your branch. Thank you.

joeyparrish commented 2 years ago

Everything we have so far in PR #1072 is building and passing all tests. If anyone wants to help write CMakeLists for additional libraries once that PR lands, I would be very happy to have the help! I think we could easily divide up the work by folder. Please let me know if you are interested!

zdanek commented 2 years ago

Joey, I had some comments to your work on your branch but I could not write it as there's no PR. Just your branch. I tried to find your email but no luck😁

Great work. I will try to help if I will have some spare time.

pon., 25 lip 2022, 20:29 użytkownik Joey Parrish @.***> napisał:

Everything we have so far in PR #1072 https://github.com/shaka-project/shaka-packager/pull/1072 is building and passing all tests. If anyone wants to help write CMakeLists for additional libraries once that PR lands, I would be very happy to have the help! I think we could easily divide up the work by folder. Please let me know if you are interested!

— Reply to this email directly, view it on GitHub https://github.com/shaka-project/shaka-packager/issues/1047#issuecomment-1194454417, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADLSJUN3JKW553ICMJLQ4TVV3MKFANCNFSM5QK5UV3A . You are receiving this because you were mentioned.Message ID: @.***>

joeyparrish commented 2 years ago

The PR is #1072, targeting the cmake branch of this main repo. Feel free to leave comments on the PR if you have feedback.

We plan to stage everything on the cmake branch, then merge that branch into main when the entire project has been ported.

If you want to discuss contributions to the CMake porting effort, let's do it here rather than by email. My inbox is insane, and I tend to lose email. Plus, as this is a public project, others might benefit from a public discussion. Thanks!

joeyparrish commented 2 years ago

If you would like to contribute to the porting effort, please reply! I get one day per week for this effort, so if I do it all by myself, it may take me months. Let's get it done together!

I have created a spreadsheet to track what is ready to port and help assign particular modules to people as we go. If you volunteer here, I will give you edit access to the sheet and add you to a chat room for porting discussions.

We currently have 3 unassigned modules with no dependencies, so you can start right away!

kevleyski commented 2 years ago

I'm going to help with this, but will check how much time I can get sponsored for vs losing my weekend :-)

vinod-balakrishnan commented 2 years ago

I can help too.. can definitely spend time on this in the weekend!!

joeyparrish commented 2 years ago

You've both been added to the spreadsheet and chat. I'll post more detailed instructions in the chat. @kevleyski, when you accept the chat invite, you should be able to see the history of what I'm about to post. If not, let me know there.

cosmin commented 2 years ago

Count me in.

Lunkers commented 2 years ago

I'd love to help! Should be able to spend a hack day or two on this at least :)

zdanek commented 2 years ago

@joeyparrish I could fix one small module just for sport ;) I've changed my work so I have no sponsor. I will do it my private time so I ask for small module for a good start. Any suggestions?

joeyparrish commented 2 years ago

@Lunkers, @zdanek, please email me (joeyparrish@google.com) and I will add you to a Google Sheet and Google Chat group to coordinate. So far, nobody has claimed any modules or started work, and there are three modules ready to be ported. When I finish media/base, many more start opening up. Thanks!

cosmin commented 2 years ago

If there is an example module done that shows the general pattern we want to follow I think it's going to be a lot easier to jump in on the other modules.

joeyparrish commented 2 years ago

Yes, definitely. If you look at the tracking spreadsheet, you'll see 5 modules are already completed: status, tools, third_party, version, file. (Those are all paths within the packager/ folder.) You can also look at the history in the cmake branch to see the diffs of the changes landing there.

zdanek commented 2 years ago

@joeyparrish I wrote an email to you. Did you get it?

cadubentzen commented 1 year ago

@joeyparrish @zdanek fyi, I started working on the CMake port as well, but haven't got write permission in the spreadsheet. Ported media/codecs. I just need to clean it up and get the CLA done to open it for review :)

zdanek commented 1 year ago

Thanks for the update. I used your codecs port as a base for my media/crypto. I did a port but one test fails. Today it's to late to fix it (I'm in CET). Point me as a reviewer I'd like to see your code when it's finished.

cosmin commented 1 year ago

The cmake branch doesn't build for me currently on OS X, because sprintf is marked as deprecated, the build is treating this warning as an error, and the cURL check for snprintf which it should use instead if detected is broken in the version we are pulling in (and every released version as of now).

https://github.com/curl/curl/pull/10155

This is the curl pull request that is necessary to get it to build. So the options I see are to update the curl submodule to latest head (until the next release is cut), or drop -Werror,-Wdeprecated-declarations to allow the deprecated sprintf to be used for now.

Romantic-LiXuefeng commented 1 year ago

Hello everyone, Thanks for the update. I want to know what's the progress now !

joeyparrish commented 1 year ago

We're getting very close to merging cmake back into main. Just a few more PRs, we think.

kevleyski commented 1 year ago

That's great! Well done and thanks all

joeyparrish commented 1 year ago

The cmake branch fully builds now! Once we clean up documentation, logging, and some automated workflows, we can merge the cmake branch into main.

gghua858 commented 1 year ago

When will the cmake branch be available? Thanks!

joeyparrish commented 1 year ago

You can check out the cmake branch now. The documentation is still out of date, but you can initialize the submodules (recursively) and build with CMake. @cosmin and I are working on finishing everything and merging back into main.

joeyparrish commented 1 year ago

Documentation in the cmake branch has been updated, and we are working on static linking, release workflows, and integration tests.

kevleyski commented 1 year ago

Great effort thanks!

M4urici0GM commented 1 year ago

I was able to build shaka-packager with cmake, but i'm getting a weird behavior: running shaka with

packager \
in=in=d16ead10625946ff81892d5980485f83.mp4,stream=audio,lang=en,roles=main,init_segment=audio/init.mp4,segment_template=audio/audio_'$Number$'.mp4 \
in=in=d16ead10625946ff81892d5980485f83.mp4,stream=video,init_segment=video/init.mp4,segment_template=video/video_'$Number$'.mp4 \
--generate_static_live_mpd \
--mpd_output manifest.mpd \

doesnt seem to generate the mpd file containing the segments:

<?xml version="1.0" encoding="UTF-8"?>
<!--Generated with https://github.com/shaka-project/shaka-packager version 2038339926-release-->
<MPD xmlns="urn:mpeg:dash:schema:mpd:2011" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="urn:mpeg:dash:schema:mpd:2011 DASH-MPD.xsd" profiles="urn:mpeg:dash:profile:isoff-live:2011" minBufferTime="PT2S" type="static" mediaPresentationDuration="PT2806.48S">
  <Period id="0">
    <AdaptationSet id="0" contentType="video" width="2560" height="1440" frameRate="90000/3600" segmentAlignment="true" par="16:9">
      <SupplementalProperty schemeIdUri="urn:mpeg:mpegB:cicp:TransferCharacteristics" value="1"/>
      <Representation id="0" bandwidth="4785100" codecs="hvc1.1.2.L150" mimeType="video/mp4" sar="1:1"/>
    </AdaptationSet>
    <AdaptationSet id="1" contentType="audio" lang="en" segmentAlignment="true">
      <Role schemeIdUri="urn:mpeg:dash:role:2011" value="main"/>
      <Representation id="1" bandwidth="167855" codecs="mp4a.40.2" mimeType="audio/mp4" audioSamplingRate="48000">
        <AudioChannelConfiguration schemeIdUri="urn:mpeg:dash:23003:3:audio_channel_configuration:2011" value="2"/>
      </Representation>
    </AdaptationSet>
  </Period>
</MPD>

may you guys check if it is reproducible on your side? Thanks

shaka-project / shaka-packager

Switch to CMake and git submodules #1047