search

mhassan1 / redis-memory-server

Redis Server for testing. The server will allow you to connect your favorite client library to the Redis Server and run parallel integration tests isolated from each other.
MIT License
76 stars 11 forks source link

Very slow installation #12

Closed Ando-git-hub closed 2 years ago

Ando-git-hub commented 2 years ago

Hi all, thanks for such a cool lib. But I have a problem, my module installation on Linux is very slow. How can I fix it?

My Node version:16.15.1 NPM version: 8.12.2

It freezes at a step that is shown in the picture bellow

Screenshot from 2022-09-05 19-38-02

But it was successfully installed and I can use it as well, my problem is very slow installation.

Thanks in advance.

mhassan1 commented 2 years ago

Unfortunately, Redis does not publish pre-compiled binaries for different systems, so this library uses make to build from source. On my machine (MacOS), it takes a few minutes, as well, but I only incur that cost on the first installation, so it's not such a problem, for me.

bigman73 commented 1 year ago

Hey Marc The problem is, unlike a MacOS, with CI/CD systems such as Github the package downloads redis server every single time the pipeline runs. There is no one time caching of the binaries.

mhassan1 commented 1 year ago

@bigman73! I think that's a general CI problem with all dependencies. All CI systems should cache node_modules and similar.

bigman73 commented 1 year ago

I guess it is, but even the initial download is heavily throttled by Github. For now I had to not use redis-memory-server, as much as I like it when it runs locally.

just an idea - what if the npm package had the compiled binary pre-built into it, for some well known architectures?

mhassan1 commented 1 year ago

It would be nice if Redis (or some other trustworthy party) built and hosted binaries for some well-known architectures. Then this package could take advantage of that.

diegoceldran commented 5 months ago

Hi,

First of all thanks for your job and this library, i think is very useful in all tests with redis.

I also has this problem with the CI/CD integration. I'm caching all the node_modules folder and it helps a few with the problem but every time i have to install another npm package or just update my dependencies i have to re-cache all the node_modules folder, making that pipeline very slow.

But my main problem is with the auto installation. My CI/CD pipelines are be able to auto install in my sandbox/production environment with docker and every time i have to build a new docker image with the updated code i have to do an npm ci/npm install and it takes so much time.

In reference to the source compilation, i'm not an expert in redis but watching the library code i can see that the default download url is this: https://download.redis.io/releases/

But if you go to https://redis.io/downloads/#redis-stack-downloads i can see a special packages for most popular distributions. For example i'm using an ubuntu 22.04 and i've downloaded this package https://packages.redis.io/redis-stack/redis-stack-server-7.2.0-v10.jammy.x86_64.tar.gz

And after i extract it i can see the redis-server binary inside bin folder.

Maybe you can check if this is enough for the redis-memory-server library?

Thanks!!

mhassan1 commented 5 months ago

@diegoceldran There are a few reasons why I don't want to do that:

  1. Redis Stack has a different license than OSS Redis (https://redis.io/legal/licenses)
  2. Redis Stack may go away or change its packaging
  3. It means maintaining mappings between OS's and package URLs, which I've seen has been costly for mongodb-memory-server (which this library is based on)

I think you have a couple options for skipping auto-install:

  1. In CI, you could cache node_modules/.cache/redis-memory-server separately, since it will change less often than the rest of node_modules
  2. In the Docker build, you could create a Docker layer that puts a pre-built redis-server binary at node_modules/.cache/redis-memory-server/redis-binaries/<version>/redis-server
diegoceldran commented 5 months ago

@diegoceldran There are a few reasons why I don't want to do that:

  1. Redis Stack has a different license than OSS Redis (https://redis.io/legal/licenses)
  2. Redis Stack may go away or change its packaging
  3. It means maintaining mappings between OS's and package URLs, which I've seen has been costly for mongodb-memory-server (which this library is based on)

I think you have a couple options for skipping auto-install:

  1. In CI, you could cache node_modules/.cache/redis-memory-server separately, since it will change less often than the rest of node_modules
  2. In the Docker build, you could create a Docker layer that puts a pre-built redis-server binary at node_modules/.cache/redis-memory-server/redis-binaries/<version>/redis-server

Thanks for the answer and the advice, I will check it.