Closed sparkyrider closed 3 years ago
There are few options:
I think option 1 would be best, however, I can use option 2 or 3 if needed.
Adding support for Compression/Decompression opens up a lot of amazing use cases for Big-data ingestion workflows, etc. when paired with AWS EFS.
Kumologica is amazing! I can't wait to see this tool become the next Unicorn company!
Hi @sparkyrider , ZIP node is now available under "Add more nodes" catalogue. Please try and share your feedback. For a guidance on the node check the help section of the ZIP node editor.
For a quick reference I am providing a sample below
[{"id":"main.flow","type":"tab","label":"main","disabled":false,"info":""},{"id":"d9d038c9.9b47e8","type":"EventListener","z":"main.flow","name":"EventListener","provider":"aws","eventSource":"api","dynamodbOperation":"","apiMethod":"get","apiUrl":"/test","albMethod":"any","albUrl":"","bucketName":"","event":"s3:ObjectCreated:*","cognitoTrigger":"any","keventSource":"","kapiMethod":"any","kapiUrl":"","kcronexpression":"","zeventSource":"","zapiMethod":"any","zapiUrl":"","x":142.5,"y":160,"wires":[["a3db3c7a.43064"]],"caname":"event-handler","category":"general"},{"id":"a8084458.dc6388","type":"EventListener-End","z":"main.flow","name":"EventListener-End","statusCode":"200","headers":{},"payload":"","x":722.5,"y":160,"wires":[],"caname":"eventlistenerend","category":"general"},{"id":"c47f34ae.5de358","type":"ZIP","z":"main.flow","name":"Compress","payload":"msg.payload","filename":"","comment":"","operation":"compress","x":342.5,"y":160,"wires":[["85cd99b3.d48c98"]],"caname":"Zip","category":"File"},{"id":"85cd99b3.d48c98","type":"FTP","z":"main.flow","name":"FTP","operation":"Put","fileloc":"/kl/testkl2.zip","payload":"msg.payload","transfermode":"ASCII","username":"","password":"","hostname":"testing","port":21,"timeout":10000,"x":482.5,"y":160,"wires":[["a8084458.dc6388"]],"caname":"FTP","category":"File"},{"id":"a3db3c7a.43064","type":"ZIP","z":"main.flow","name":"Add the file","payload":"'Hello Kumologica zip'","filename":"kl.txt","comment":"testing","operation":"addfile","x":262.5,"y":160,"wires":[["c47f34ae.5de358"]],"caname":"Zip","category":"File"}]
Note : Install ZIP node and FTP node from "Add more nodes" to use the above given sample.
Earlier sample was for zipping a text content and pushing to FTP location. The below is a sample where a zip having a text file is read from an FTP location and extracted using the ZIP node. The extracted content is printed using the logger node.
[{"id":"main.flow","type":"tab","label":"main","disabled":false,"info":""},{"id":"d9d038c9.9b47e8","type":"EventListener","z":"main.flow","name":"EventListener","provider":"aws","eventSource":"api","dynamodbOperation":"","apiMethod":"get","apiUrl":"/test","albMethod":"any","albUrl":"","bucketName":"","event":"s3:ObjectCreated:*","cognitoTrigger":"any","keventSource":"","kapiMethod":"any","kapiUrl":"","kcronexpression":"","zeventSource":"","zapiMethod":"any","zapiUrl":"","x":142.5,"y":160,"wires":[["85cd99b3.d48c98"]],"caname":"event-handler","category":"general"},{"id":"a8084458.dc6388","type":"EventListener-End","z":"main.flow","name":"EventListener-End","statusCode":"200","headers":{},"payload":"","x":722.5,"y":160,"wires":[],"caname":"eventlistenerend","category":"general"},{"id":"c47f34ae.5de358","type":"ZIP","z":"main.flow","name":"Extract","payload":"msg.payload","filename":"","comment":"","operation":"extract","x":382.5,"y":160,"wires":[["f0cb9c6c.e619c"]],"caname":"Zip","category":"File"},{"id":"85cd99b3.d48c98","type":"FTP","z":"main.flow","name":"FTP","operation":"Get","fileloc":"/kl/testkl2.zip","payload":"msg.payload","transfermode":"ASCII","username":"","password":"","hostname":"testing","port":21,"timeout":10000,"x":282.5,"y":160,"wires":[["c47f34ae.5de358"]],"caname":"FTP","category":"File"},{"id":"f0cb9c6c.e619c","type":"Function","z":"main.flow","name":"BufferToString","noerr":0,"query":"// Available objects in the context: msg, vars, env\n// Logging: node.log|info|warn|debug|error(\"message\")\n// Errors: throw new Error('message') \n\nlet buff = msg.payload[0].content\nmsg.payload = buff.toString();\n\nreturn msg;\n","x":502.5,"y":160,"wires":[["61c65717.b8b188"]],"caname":"function","category":"transformation"},{"id":"61c65717.b8b188","type":"Logger","z":"main.flow","name":"Logger","level":"INFO","message":"msg.payload","format":"string","headers":{},"x":622.5,"y":160,"wires":[["a8084458.dc6388"]],"caname":"logger","category":"logging"}]
12 Aug 14:20:37 - [ warn] Communication send error: RangeError: Invalid string length
For running a test case locally, is there a RAM limit setting that I can set or is it using w/e the local computer has available? I think the Compressed file is ~9MB Compressed, 60MB uncompressed.
@ab73863 I was able to get it working on very small Zip files, however, even zip files uncompressed around 5-8mb fails with the same error. Is there a Debug Log somewhere I can check/share?
@sparkyrider The error [ warn] Communication send error: RangeError: Invalid string length
which you are seeing is due to a bug in the designer test tab. This is not an error due to any limitation of unzipping by the zip node.
From runtime perspective the zip file that you have provided should be unzipped by the zip node and is available on memory.
To check this you can actually try putting a logger after the ZIP node (post extraction) with the following on the Message : 'No of files extracted : ' & $count(msg.payload)
. This will print the number of files that got extracted and confirms the extraction was
fine.
The output of extraction will be an array containing object having extracted file and its content as buffer. eg : [ {"filename" : "file1.png", "content" : [buffercontent] }, {"filename" : "file2.pdf", "content" : [buffercontent] }, {"filename" : "file3.txt", "content" : [buffercontent] } ]
The fix for the designer tab issue will be made available in our next release. And as I mentioned earlier you could still extract larger zip file on lambda without any issues. Will keep you posted on the fix update.
@sparkyrider The patch for the designer covering the fix will be available next week.
Zip node works great, looking forward to the Designer update. Thanks so much!
Is it possible to unzip a file from a zip file with Kumologica?