search

Blizzard / node-rdkafka

Node.js bindings for librdkafka
MIT License
2.1k stars 390 forks source link

Major difference in size between v2.14.0 and v.15.0 #1030

Open andreea96 opened 1 year ago

andreea96 commented 1 year ago

Environment Information

Steps to Reproduce when upgrading the library from v2.13 to v2.15 we were unable to deploy to lambda as the size of the package increased from 150MB to 340 MB. The reason we need this update: when using version 2.13 we get the following error on publishing:

Unsupported value "SASL_SSL" for configuration property "security.protocol": OpenSSL not available at build time

with the following dockerfile config (where the npm ci is taking place): 


FROM public.ecr.aws/lambda/nodejs:16.2022.05.23.09

RUN yum update -y && \
    yum install -y \
    zip \
    bash \
    python3 \
    g++ \
    gcc \
    libgcc-5-dev \
    libstdc++6 \
    linux-headers-$(uname -r) \
    gcc-c++ \
    make \
    openssl-dev \
    openssl \
    cyrus-sasl-dev \
    which \
    ca-certificates \
    lz4-dev \
    musl-dev 

ENTRYPOINT []

and producer config for auth looks like this: 

{
                        'security.protocol': 'sasl_ssl' as const,
                        'sasl.mechanism': 'SCRAM-SHA-512',
                        'sasl.username': 'user',
                        'sasl.password': 'pass'
}

I have noticed there was an update of the librdkafka to v2.0.2 in : https://github.com/Blizzard/node-rdkafka/releases/tag/v2.15.0 so i suspect this is what may have fixed this openssl issue, how could we go around this issue while still being able to upload to lambda. The package is so large that i am unable to upload it neither as a lambda layer.

andreea96 commented 1 year ago

After some digging i found out the package increase is from de deps folder that contains the librdkafka files, is this folder necessary during runtime or can it be removed after installation?

tatethurston commented 11 months ago

Also encountering this for lambda deployments. I haven't had a chance to dig in yet, but at a glance it looks like the files copied over during build could have expanded:

https://github.com/Blizzard/node-rdkafka/pull/996/files#diff-ca009902e3b4fed89eada60c34e9ae9d3120cf9b1cf6e806d689ed19ea05da86R69

It looks like the node_module could be paired down considerably, it would be great to consider holistically pairing things down with .npmignore and only preserving the necessary artifacts following the node-gyp build.