let4be / crusty

Broad Web Crawler
GNU General Public License v3.0
83 stars 3 forks source link

Error in redis dockerfile #39

Closed ErickJ3 closed 10 months ago

ErickJ3 commented 11 months ago

Hello, I hope you are well. I've been trying to run the project locally on my Docker, but I always get an error in the redis dockerfile. I updated some things in the docker file, but the problem still persists:

FROM redis

# Update and install necessary packages
RUN apt-get update && apt-get -y install git build-essential cmake

# Create a symlink for python3 to ensure it is recognized

# Create app directory and clone RedisBloom
RUN mkdir /app && cd /app && \
    git clone https://github.com/RedisBloom/RedisBloom && \
    cd RedisBloom && \
    git submodule update --init --recursive && \
    ./sbin/setup \
    bash -l \
    make

# Copy configuration and modules
COPY redis.conf /usr/local/etc/redis/redis.conf
COPY --from=crusty_crusty:latest /usr/local/lib/libredis_queue.so /app
COPY --from=crusty_crusty:latest /usr/local/lib/libredis_calc.so /app

# Expose port
EXPOSE 6379/tcp
CMD [ "redis-server", "/usr/local/etc/redis/redis.conf", "--loadmodule /app/libredis_queue.so", "--loadmodule /app/libredis_calc.so", "--loadmodule /app/RedisBloom/redisbloom.so" ]

error that is happening:

2023-11-27 16:51:55 1:M 27 Nov 2023 19:51:55.351 * Module 'crusty.queue' loaded from /app/libredis_queue.so
2023-11-27 16:51:55 1:M 27 Nov 2023 19:51:55.351 * Module 'crusty.calc' loaded from /app/libredis_calc.so
2023-11-27 16:51:55 1:M 27 Nov 2023 19:51:55.351 # Module /app/RedisBloom/redisbloom.so failed to load: /app/RedisBloom/redisbloom.so: cannot open shared object file: No such file or directory
2023-11-27 16:51:55 1:M 27 Nov 2023 19:51:55.351 # Can't load module from /app/RedisBloom/redisbloom.so: server aborting
let4be commented 11 months ago

Hi!

This project uses redisbloom module for redis, looks like for some reason it cannot find it(it didn't build)

this line in Dockerfile RUN mkdir /app && cd /app && git clone https://github.com/RedisBloom/RedisBloom && cd RedisBloom && git submodule update --init --recursive && make supposed to take care of it

probably there's something off with the build or I should have pinned the version of RedisBloom(likely the cause) Version of redis is also pinned for a reason of reproducible builds.

I will take a look on the weekends hopefully :)

ErickJ3 commented 10 months ago

Bloom is not compiling correctly in the dockerfile environment, but I decided to compile redisbloom and copy it to the container where it would be used.

Do you intend to continue with the project? Maybe store more data from the sites being crawled and index them in elasticsearch or apache sorl?

@let4be

let4be commented 10 months ago

No time for this project right now unfortunately, busy with other stuff.

Elastic or anything else can be easily connected, I think(the data is here just being discarded)