Can't build media-scanner

dimitry-ishenko commented 1 month ago

Trying to follow build instructions in README.md...

yarn install runs fine:

root@laptop:/build/media-scanner# yarnpkg install
yarn install v1.22.19
[1/5] Validating package.json...
[2/5] Resolving packages...
warning @sofie-automation/code-standard-preset > eslint > @humanwhocodes/config-array@0.13.0: Use @eslint/config-array instead
warning @sofie-automation/code-standard-preset > license-checker > read-installed@4.0.3: This package is no longer supported.
warning @sofie-automation/code-standard-preset > eslint > @humanwhocodes/config-array > @humanwhocodes/object-schema@2.0.3: Use @eslint/object-schema instead
warning @sofie-automation/code-standard-preset > license-checker > nopt > osenv@0.1.5: This package is no longer supported.
warning @sofie-automation/code-standard-preset > license-checker > read-installed > debuglog@1.0.1: Package no longer supported. Contact Support at https://www.npmjs.com/support for more info.
warning @sofie-automation/code-standard-preset > license-checker > read-installed > read-package-json@2.1.2: This package is no longer supported. Please use @npmcli/package-json instead.
warning @sofie-automation/code-standard-preset > license-checker > read-installed > readdir-scoped-modules@1.1.0: This functionality has been moved to @npmcli/fs
warning @sofie-automation/code-standard-preset > license-checker > read-installed > readdir-scoped-modules > debuglog@1.0.1: Package no longer supported. Contact Support at https://www.npmjs.com/support for more info.
warning @sofie-automation/code-standard-preset > eslint > file-entry-cache > flat-cache > rimraf@3.0.2: Rimraf versions prior to v4 are no longer supported
warning @sofie-automation/code-standard-preset > license-checker > read-installed > read-package-json > glob@7.2.3: Glob versions prior to v9 are no longer supported
warning @sofie-automation/code-standard-preset > eslint > file-entry-cache > flat-cache > rimraf > glob@7.2.3: Glob versions prior to v9 are no longer supported
warning @sofie-automation/code-standard-preset > license-checker > read-installed > read-package-json > glob > inflight@1.0.6: This module is not supported, and leaks memory. Do not use it. Check out lru-cache if you want a good and tested way to coalesce async requests by a key value, which is much more comprehensive and powerful.
warning jest > @jest/core > jest-runtime > glob@7.2.3: Glob versions prior to v9 are no longer supported
warning jest > @jest/core > @jest/reporters > glob@7.2.3: Glob versions prior to v9 are no longer supported
warning jest > @jest/core > jest-config > glob@7.2.3: Glob versions prior to v9 are no longer supported
warning jest > @jest/core > @jest/transform > babel-plugin-istanbul > test-exclude > glob@7.2.3: Glob versions prior to v9 are no longer supported
warning zip-a-folder > archiver > archiver-utils > glob@7.2.3: Glob versions prior to v9 are no longer supported
warning zip-a-folder > archiver > zip-stream > archiver-utils > glob@7.2.3: Glob versions prior to v9 are no longer supported
[3/5] Fetching packages...
[4/5] Linking dependencies...
[5/5] Building fresh packages...
success Saved lockfile.
$ husky
Done in 39.39s.

yarn build give this error:

root@laptop:/build/media-scanner# yarnpkg build
yarn run v1.22.19
$ run build:ts && node tools/build.mjs
/bin/sh: 1: run: not found
error Command failed with exit code 127.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

yarn build-linux-x64 gives this error:

root@laptop:/build/media-scanner# yarnpkg build-linux-x64
yarn run v1.22.19
$ node tools/build.mjs linux x64
Building for linux-x64
> pkg@5.8.1
> Error! Input file does not exist
  /build/media-scanner/dist/index.js

error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Where is dist/index.js supposed to come from?

dimitry-ishenko commented 1 month ago

npm gives basically the same errors:

root@laptop:/build/media-scanner# npm run build

> scanner@1.3.4 build
> run build:ts && node tools/build.mjs

sh: 1: run: not found
root@laptop:/build/media-scanner#

root@laptop:/build/media-scanner# npm run build-linux-x64

> scanner@1.3.4 build-linux-x64
> node tools/build.mjs linux x64

Building for linux-x64
> pkg@5.8.1
> Error! Input file does not exist
  /build/media-scanner/dist/index.js

It looks like the build script is just broken, but the build-linux-x64 one is missing a file of some sort?

dimitry-ishenko commented 1 month ago

The Dockerfile script is likewise broken:

dimitry@laptop:~/casparcg/media-scanner$ docker build -t scanner .
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            Install the buildx component to build images with BuildKit:
            https://docs.docker.com/go/buildx/

Sending build context to Docker daemon  16.41MB
Step 1/11 : FROM node:18
 ---> b92b80169bfd
Step 2/11 : WORKDIR /usr/src/app
 ---> Using cache
 ---> 6489748920ac
Step 3/11 : ENV NODE_ENV production
 ---> Using cache
 ---> 09409b90c326
Step 4/11 : ENV PATHS__FFMPEG ffmpeg
 ---> Using cache
 ---> 198f253051e8
Step 5/11 : ENV PATHS__FFPROBE ffmpeg
 ---> Using cache
 ---> c76c0ecd411c
Step 6/11 : COPY package.json package-lock.json ./
COPY failed: file not found in build context or excluded by .dockerignore: stat package-lock.json: file does not exist

package-lock.json file doesn't exist in the repo...

I've removed package-lock.json from Dockerfile and tried again:

dimitry@laptop:~/casparcg/media-scanner$ docker build -t scanner .
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            Install the buildx component to build images with BuildKit:
            https://docs.docker.com/go/buildx/

Sending build context to Docker daemon  16.53MB
Step 1/11 : FROM node:18
 ---> b92b80169bfd
Step 2/11 : WORKDIR /usr/src/app
 ---> Using cache
 ---> 6489748920ac
Step 3/11 : ENV NODE_ENV production
 ---> Using cache
 ---> 09409b90c326
Step 4/11 : ENV PATHS__FFMPEG ffmpeg
 ---> Using cache
 ---> 198f253051e8
Step 5/11 : ENV PATHS__FFPROBE ffmpeg
 ---> Using cache
 ---> c76c0ecd411c
Step 6/11 : COPY package.json ./
 ---> b74e072190b0
Step 7/11 : RUN sed -i -e 's/^  "version": "[0-9.]\+",$//' package.json
 ---> Running in 03b050bfeef5
Removing intermediate container 03b050bfeef5
 ---> 3ad9b8b72f63
Step 8/11 : RUN yarn install &&       apt-get update &&       apt-get install ffmpeg -y
 ---> Running in be0262b8aa4d
yarn install v1.22.19
info No lockfile found.
[1/5] Validating package.json...
[2/5] Resolving packages...
warning @sofie-automation/code-standard-preset > eslint > @humanwhocodes/config-array@0.13.0: Use @eslint/config-array instead
warning @sofie-automation/code-standard-preset > license-checker > read-installed@4.0.3: This package is no longer supported.
warning @sofie-automation/code-standard-preset > eslint > @humanwhocodes/config-array > @humanwhocodes/object-schema@2.0.3: Use @eslint/object-schema instead
warning @sofie-automation/code-standard-preset > license-checker > read-installed > debuglog@1.0.1: Package no longer supported. Contact Support at https://www.npmjs.com/support for more info.
warning @sofie-automation/code-standard-preset > license-checker > read-installed > read-package-json@2.1.2: This package is no longer supported. Please use @npmcli/package-json instead.
warning @sofie-automation/code-standard-preset > license-checker > read-installed > readdir-scoped-modules@1.1.0: This functionality has been moved to @npmcli/fs
warning @sofie-automation/code-standard-preset > license-checker > read-installed > readdir-scoped-modules > debuglog@1.0.1: Package no longer supported. Contact Support at https://www.npmjs.com/support for more info.
warning @sofie-automation/code-standard-preset > license-checker > nopt > osenv@0.1.5: This package is no longer supported.
warning @sofie-automation/code-standard-preset > eslint > file-entry-cache > flat-cache > rimraf@3.0.2: Rimraf versions prior to v4 are no longer supported
warning @sofie-automation/code-standard-preset > license-checker > read-installed > read-package-json > glob@7.2.3: Glob versions prior to v9 are no longer supported
warning @sofie-automation/code-standard-preset > eslint > file-entry-cache > flat-cache > rimraf > glob@7.2.3: Glob versions prior to v9 are no longer supported
warning @sofie-automation/code-standard-preset > license-checker > read-installed > read-package-json > glob > inflight@1.0.6: This module is not supported, and leaks memory. Do not use it. Check out lru-cache if you want a good and tested way to coalesce async requests by a key value, which is much more comprehensive and powerful.
warning jest > @jest/core > jest-config > glob@7.2.3: Glob versions prior to v9 are no longer supported
warning jest > @jest/core > jest-runtime > glob@7.2.3: Glob versions prior to v9 are no longer supported
warning jest > @jest/core > @jest/reporters > glob@7.2.3: Glob versions prior to v9 are no longer supported
warning jest > @jest/core > @jest/transform > babel-plugin-istanbul > test-exclude > glob@7.2.3: Glob versions prior to v9 are no longer supported
warning zip-a-folder > archiver > archiver-utils > glob@7.2.3: Glob versions prior to v9 are no longer supported
warning zip-a-folder > archiver > zip-stream > archiver-utils > glob@7.2.3: Glob versions prior to v9 are no longer supported
[3/5] Fetching packages...
[4/5] Linking dependencies...
[5/5] Building fresh packages...
success Saved lockfile.
$ husky
/bin/sh: 1: husky: not found
error Command failed with exit code 127.
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
The command '/bin/sh -c yarn install &&       apt-get update &&       apt-get install ffmpeg -y' returned a non-zero code: 127

dimitry-ishenko commented 1 month ago

Just for testing decided to run yarn dev and got this:

root@laptop:/build/media-scanner# yarnpkg dev
yarn run v1.22.19
$ nodemon ./src
[nodemon] 2.0.22
[nodemon] to restart at any time, enter `rs`
[nodemon] watching path(s): src/**/*
[nodemon] watching extensions: ts
[nodemon] starting `ts-node src/index.ts ./src`
/build/media-scanner/node_modules/ts-node/src/index.ts:859
    return new TSError(diagnosticText, diagnosticCodes, diagnostics);
           ^
TSError: ⨯ Unable to compile TypeScript:
src/util.ts:3:28 - error TS2305: Module '"cheerio"' has no exported member 'Element'.

3 import cheerio, { Cheerio, Element } from 'cheerio'
                             ~~~~~~~

    at createTSError (/build/media-scanner/node_modules/ts-node/src/index.ts:859:12)
    at reportTSError (/build/media-scanner/node_modules/ts-node/src/index.ts:863:19)
    at getOutput (/build/media-scanner/node_modules/ts-node/src/index.ts:1077:36)
    at Object.compile (/build/media-scanner/node_modules/ts-node/src/index.ts:1433:41)
    at Module.m._compile (/build/media-scanner/node_modules/ts-node/src/index.ts:1617:30)
    at Module._extensions..js (node:internal/modules/cjs/loader:1414:10)
    at Object.require.extensions.<computed> [as .ts] (/build/media-scanner/node_modules/ts-node/src/index.ts:1621:12)
    at Module.load (node:internal/modules/cjs/loader:1197:32)
    at Function.Module._load (node:internal/modules/cjs/loader:1013:12)
    at Module.require (node:internal/modules/cjs/loader:1225:19) {
  diagnosticCodes: [ 2305 ]
}
[nodemon] app crashed - waiting for file changes before starting...

dimitry-ishenko commented 1 month ago

I dunno if I am doing something wrong... or this project is just an utterly broken mess... 😠 😠

nytamin commented 1 month ago

There likely is an issue with your local environment, as the builds work fine in the CI env.

I see in your logs that you are using the classic yarn version 1, this might be the issue. Try uninstalling it and install yarn 2+.

If you're still not able to figure out what's wrong with your environment, try replacing the run statements in the scripts in package.json with yarn, that should do the trick for you locally.

dimitry-ishenko commented 1 month ago

I see in your logs that you are using the classic yarn version 1, this might be the issue. Try uninstalling it and install yarn 2+.

Thanks @nytamin. I've no idea how yarn versions work, but here are my versions:

root@laptop:/build/media-scanner# node --version
v18.19.1
root@laptop:/build/media-scanner# npm --version
9.2.0
root@laptop:/build/media-scanner# yarnpkg --version
1.22.19

compared to the versions used in the CI:

Screenshot from 2024-09-26 10-29-47

My versions are slightly behind, but I don't see yarn 2 anywhere...

dimitry-ishenko commented 1 month ago

Oh, so whatever environment you have doesn't matter since, corepack enable installs yarn 4.0.2... So, let me get this straight:

In order to install corepack, you need to have npm installed...
Then, you need to run npm install corepack to install corepack...
Then, you need to run corepack enable to install yarn...
The above doesn't actually install new version of yarn, but adds some sort of a hook...
Then, you need to have an older version of yarn installed...
Then, you need to run yarn install in order to install new version of yarn...
And only then, you can run yarn build to build the package

I am sorry, but whoever came up with this mess should be lynched.

dimitry-ishenko commented 1 month ago

But the story doesn't end there. Now I get the same error as with yarn dev:

root@laptop:/build/media-scanner# yarn build
src/util.ts:3:28 - error TS2305: Module '"cheerio"' has no exported member 'Element'.

3 import cheerio, { Cheerio, Element } from 'cheerio'
                             ~~~~~~~

Found 1 error.

root@laptop:/build/media-scanner#

Mind blown... 🤯 🤯 🤯

Julusian commented 1 month ago

In order to install corepack, you need to have npm installed...

It used to be that corepack was bundled with node. I didn't think they had changed that, but there has been some recent rumblings that it would be.

Then, you need to have an older version of yarn installed...

No, corepack will fetch whatever version of yarn is needed. It installs a yarn executable which does this and then calls that version.

So the steps (last I checked) are:

Install node
Then, you need to run corepack enable to install yarn...
Then, you need to run yarn install in order to install new version of yarn...
And only then, you can run yarn build to build the package

But I've probably neglected the docker build, causing some errors there

Julusian commented 1 month ago

Oh that dockerfile is entirely broken. Not been updated to handle that the code is typescript or anything

dimitry-ishenko commented 1 month ago

It used to be that corepack was bundled with node. I didn't think they had changed that, but there has been some recent rumblings that it would be.

Well, neither Debian nor Ubuntu bundle corepack with NodeJS. I guess because it's still considered experimental. There is node-corepack package in Debian unstable/testing and the upcoming Ubuntu version... but that doesn't really help. Debian packages don't usually allow downloading of external dependencies during build process.

Julusian commented 1 month ago

The docker image now builds. I only tested it as far as it complained it couldnt find the config file.

Debian packages don't usually allow downloading of external dependencies during build process.

Surely that means that yarn install is not allowed either? ;)

Tbh I have little interest in supporting whatever version of nodejs ubuntu/debian package. Ensuring this works with the node 12 that is available in ubuntu 22.04 is going to be a painful task, and I don't want to be held back on node 18 either. Doing so means essentially not updating or adding any dependencies, as while they generally follow semver, some don't care if they publish a semver minor/patch version that breaks support for eol nodejs.

This is why for https://github.com/Julusian/CasparCG-ppa/tree/main I simply take the built executables from here. I probably should at least make that use a folder of code rather than the executable, but it works and hasn't bothered me enough to change it

dimitry-ishenko commented 1 month ago

Surely that means that yarn install is not allowed either? ;)

That's right. Debian/Ubuntu provides a lot of node-* packages in their repo and dh-nodejs script which helps pull in needed deps using the package management system... But, I haven't crossed that bridge yet.

dimitry-ishenko commented 1 month ago

Tbh I have little interest in supporting whatever version of nodejs ubuntu/debian package. Ensuring this works with the node 12 that is available in ubuntu 22.04 is going to be a painful task, and I don't want to be held back on node 18 either.

@Julusian that's why we should re-write media-scanner using a normal language like C++. IMHO.

dimitry-ishenko commented 1 month ago

Surely that means that yarn install is not allowed either? ;)

FWIW here is an excerpt from the Debian Policy manual, section 4.9:

For packages in the main archive, required targets must not attempt network access, except, via the loopback interface, to services on the build host that have been started by the build.

Julusian commented 1 month ago

@Julusian that's why we should re-write media-scanner using a normal language like C++. IMHO.

I'm not convinced that C++ would be the best fit here. The scanner became its own thing because the previous implementation inside casparcg was buggy and crash prone. While a fresh implementation might be better, it also might not.

I think that this project, seeing as it doesn't need absolute maximum performance, is better suited to something more developer and contributor friendly.
I can say with reasonable certainty that the changes I contributed through work for the /media and related endpoints wouldn't have happened if this was c++. I expect that we would have instead made our own nodejs based scanning tool to do this.

Python is a more accepted language in linux circles, and I would be open to using that. Problem is that my python knowledge is basically zero, so making that change would greatly decrease my ability and motivation to maintain this. I don't what typings are like in python by now, I find that js without types is hard to follow and very easy to make mistakes in.

Also you've seen the poc I did for the server which ports the control plane to nodejs. I want less C++ to deal with in areas where it is avoidable.

CasparCG / media-scanner

Can't build media-scanner #86