Allow runtime environment definition in `actor.json` instead of using Dockerfile

[mnmkng]

I've been thinking about this for quite some time. Our Dockerfiles are becoming increasingly more complex. The last complexities come from multistage builds with TypeScript. There's always something to fix and it's an ever growing pain to manage.

⚠️ I'm not saying that we should replace Dockerfiles. They need to stay, but...

Most of our actors use one of our predefined base images AND predefined Dockerfile templates. In most cases, users shouldn't need to change those at all. Issues like this or this happen repeatedly for years.

Proposal:

Allow specifying the runtime in actor.json instead of in Dockerfile. E.g.

{
    "environment": {
        "engine": "node:16", // or python
        "browsers": ["chrome"]
    }
}

Or, although I would not suggest this, but it could be useful for quickly running new projects.

{
    "environment": "auto"
}

This would detect the env based on presence of package.json or some Python files and then analyzing the package.json for dependencies. Basically something like we had for single file, but instead of parsing code, it would parse package.json.

Possible benefits:

• Users wouldn't have to worry about Dockerfiles. It's a very low level thing for many of them.
• We would reduce the surface area for things to break.
• We could reduce the number of images on the platform and we could make performance optimizations in how the images are loaded and built. In many cases, people don't use any external deps besides SDK and Crawlee (with Playwright and so on...), so maybe we wouldn't even have to run npm install in most cases and we could just copy the source and be done with it.

What do you think? @jancurn @mtrunkat @fnesveda @dragonraid @B4nan

[B4nan]

Few random thoughts:

• I like the idea of simplification for users.
• We could have some "eject dockerfile" command in CLI to allow overriding, like the CRA and ejecting webpack config.
• But I think most of the issues come from the base docker images pre-installing things, I don't see how hiding dockerfiles would solve it, the same problem would remain, you still use a dockerfile, it would be the same as in the current templates, with same preinstalled dependencies.
• The latest issue is really just about a missing step in pipeline, the docker images were not rebuild, the proposal would not affect it anyhow - the cache takes precedence even when we run npm install inside the image
• This would be applicable only for the Apify platform and CLI, crawlee templates would still use the explicit dockerfiles I guess?

So overall, why not, I just wouldn't expect it will solve much of our current problems, rather it will simplify things for users (which is always nice), but only for those that will go with the Apify CLI right ahead - for crawlee users that will try to deploy on the platform it won't change anything.

[mnmkng]

Yeah, in a way I agree with the comments. But I think we could have better insight into what's actually happening in our images if we were only copying source into them as the last step instead of doing the whole pre-install (in base dockers) + re-install (in userland) magic that we're doing now. Basically, we would control a larger part of the build process.

[mtrunkat]

I see the problem, but I am afraid that by hiding this complexity, we get into very similar problems as we had with single file - i.e.

• something will break between builds because it's not in the code
• or users won't be able to debug certain problems.

As a plus, I see that this way, we could have been able to limit the number of image variations and so the size of builds and increase the performance of workers.

I'd first consider more solutions to this - maybe we can somehow improve the images or bring better defaults?

[jancurn]

Closing this for inactivity