Increased memory usage on google cloud functions on Node 8?

[ovaris]

Hi,

After upgrading from Node 6 to 8 (Beta support in google cloud functions atm) I noticed memory increase in memory usage. I have a function that overlays 10-20 images using read and write streams.

Heres mem drop after downgrading back to Node 6 engine:

This is not a bug report, more like a question that is there something I need to be aware of or anything I could do to decrease mem usage? Of course bug also could be on Google's side.

[lovell]

My best guess would be that the change in runtime from Node 6 to 8 also involved OS-level changes, perhaps relating to memory allocator configuration. The Node 6 runtime is "based on Debian 8" whereas the Node 8 runtime is "based on Ubuntu 18.04" (which is based on Debian 10).

If the "memory usage" graph shows peak RSS then remember this includes free memory that has yet to be returned to the OS. It looks like you've provisioned 2GB. What happens if you provision the functions to only have access to 1GB?

Perhaps with the Node 8 runtime Google have tuned the OS to not demand free memory is returned, or tuned the memory allocator to not return free memory in an attempt to improve (perceived) performance?

[lovell]

@ovaris Were you able to make any progress with this, perhaps with information from Google?

[ovaris]

Sorry @lovell I have not actually, been to busy and just stuck with Node 6 version since it seems to be running with lower memory. Getting info from Google is painfully slow process unfortunately. This function started to act weird after upgrading to Node 8 ( it only saved partial data into real time database and leaving some fields nil), but I really don't know if it was due to function running on memory limits, or was it because of some other problem in beta Node 8 runtime.

[lovell]

@ovaris Thanks for the update. I'd be happy to alert some Google people to this issue if you have the time to continue to diagnose the problem with them.

[ovaris]

Yeah sure, please give heads up for Google people. We need to upgrade anyways eventually to Node 8

[lovell]

If possible, please can you upgrade sharp to the new v0.21.0 version first and confirm this remains a problem.

[ovaris]

Sorry for the delay, I'm upgrading right now to v0.21.0 (need to go through the code to fix deprecated calls), then my plan is to deploy to production. First I will see how updated lib performs with Node 6, then I can switch engine to 8 from cloud console and see what happens.

[ovaris]

Looking good so far! Updated at where blue dot is (22:30):

[ovaris]

Hey, looks like this has been now fixed, either a) by updating Sharp or b) They've fixed something in cloud functions, env. Memory usage after upgrade (blue dot):

[lovell]

Thanks for the updates, glad to hear it's now working as expected. It would be interesting to see if downgrading sharp but keeping the newer version of Node has any effect.

[ovaris]

@lovell I'm not sure but I think I'm seeing increased memory usage after all, which I didn't see with Node 6, I can see even some 2GB peaks, here's 30 day graph,

[ovaris]

Here's the drop after downgrading:

[DavidKuennen]

Having the same Problem with sharp 0.21.3

But only when I use toBuffer or pipe

[himadrinath]

@lovell @DavidKuennen @ovaris same problem for me too i config vm with 1gb memory and using sharp 0.21.3 when i processing image one at at time i saw huge increase in memeory and after that -bash: fork: Cannot allocate memory

<--- JS stacktrace ---> Cannot get stack trace in GC. FATAL ERROR: NewSpace::Rebalance Allocation failed - process out of memory 1: node::Abort() [node] 2: 0x8cd14c [node] 3: v8::Utils::ReportOOMFailure(char const*, bool) [node] 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [node] 5: 0xa6717b [node] 6: v8::internal::MarkCompactCollector::Evacuate() [node] 7: v8::internal::MarkCompactCollector::CollectGarbage() [node] 8: v8::internal::Heap::MarkCompact() [node] 9: v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [node] 10: v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node] 11: v8::internal::Factory::NewMap(v8::internal::InstanceType, int, v8::internal::ElementsKind) [node] 12: v8::internal::Map::RawCopy(v8::internal::Handle<v8::internal::Map>, int) [node] 13: v8::internal::Map::CopyDropDescriptors(v8::internal::Handle<v8::internal::Map>) [node] 14: v8::internal::Map::ShareDescriptor(v8::internal::Handle<v8::internal::Map>, v8::internal::Handle<v8::internal::DescriptorArray>, v8::internal::Descriptor*) [node] 15: v8::internal::Map::CopyAddDescriptor(v8::internal::Handle<v8::internal::Map>, v8::internal::Descriptor*, v8::internal::TransitionFlag) [node] 16: v8::internal::Map::CopyWithField(v8::internal::Handle<v8::internal::Map>, v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::FieldType>, v8::internal::PropertyAttributes, v8::internal::PropertyConstness, v8::internal::Representation, v8::internal::TransitionFlag) [node] 17: v8::internal::Map::TransitionToDataProperty(v8::internal::Handle<v8::internal::Map>, v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes, v8::internal::PropertyConstness, v8::internal::Object::StoreFromKeyed, bool*) [node] 18: v8::internal::LookupIterator::PrepareTransitionToDataProperty(v8::internal::Handle<v8::internal::JSObject>, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes, v8::internal::Object::StoreFromKeyed) [node] 19: v8::internal::Object::AddDataProperty(v8::internal::LookupIterator*, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes, v8::internal::Object::ShouldThrow, v8::internal::Object::StoreFromKeyed) [node] 20: v8::internal::JSObject::DefineOwnPropertyIgnoreAttributes(v8::internal::LookupIterator*, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes, v8::internal::Object::ShouldThrow, v8::internal::JSObject::AccessorInfoHandling) [node] 21: v8::internal::JSObject::DefinePropertyOrElementIgnoreAttributes(v8::internal::Handle<v8::internal::JSObject>, v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes) [node] 22: v8::internal::JsonParser<true>::ParseJsonObject() [node] 23: v8::internal::JsonParser<true>::ParseJsonValue() [node] 24: v8::internal::JsonParser<true>::ParseJson() [node] 25: v8::internal::Builtin_JsonParse(int, v8::internal::Object**, v8::internal::Isolate*) [node] 26: 0x380ccc98697d Aborted (core dumped)