feathersjs-ecosystem / feathers-sync

Synchronize service events between Feathers application instances
MIT License
221 stars 41 forks source link

High RAM usage when large files are uploaded #181

Closed green3g closed 2 years ago

green3g commented 2 years ago

Steps to reproduce

(First please check that this issue is not already solved as described here)

I am running a file service that has been sort of based off of the example file uploads in the feathers docs. When I upload a file that is fairly large ( > 100 MB ) the service starts using many GB of RAM and eventually crashes. Removing Feathers Sync from the app, the files are processed/uploaded immediately.

Attached is my files service (with hooks):

files.zip

Required config:

# config/default.yml
blobStore:
  type: local
  path: ../data

Expected behavior

Feathers-sync shouldn't cause large amounts of RAM usage. I think there may be an issue with how the events are published with large files.

Actual behavior

I don't know exactly where the problem in this lies - it seems that when a request gets published that has a large file in it, the feathers-sync appears to use a lot of RAM.

System configuration

Tell us about the applicable parts of your setup.

Module versions (especially the part that's not working): ❯ npm ls feathers-sync wsb-services-ts@1.16.2 /Users/groemhildt/projects/wsb-services-ts └── feathers-sync@3.0.1

❯ npm ls @feathersjs/feathers wsb-services-ts@1.16.2 /Users/groemhildt/projects/wsb-services-ts ├─┬ @feathersjs/authentication-client@4.5.13 │ └── @feathersjs/feathers@4.5.12 deduped

NodeJS version: ❯ node --version v16.13.1

Operating System: MacOS 12.2.1 Monteray

Browser Version: Firefox 98.0.2

Module Loader: webpack

daffl commented 2 years ago

You can disable synchronization for the blob service or set context.event to null to emitting the event.

green3g commented 2 years ago

Awesome. Yes that'll work for me. Thank you @daffl !