search

radiegtya / meteoris2

a Realtime Javascript Boilerplate base on Meteor Js Framework
MIT License
246 stars 45 forks source link

Post Processing (Workflow / MQ) #5

Closed KrishnaPG closed 9 years ago

KrishnaPG commented 9 years ago

First of all thank you much for this very helpful package. Proves invaluable for many beginners like me.

Been testing the demo project and working through the code to understand it. I have few followup questions. Request you to kindly spare few moments to give your valuable pointers:

  1. Can this package be used to build production grade systems or is this only for educational purposes?
  2. If the answer to the above questions is 'yes', then followup question: In the posts section, after upload, I would like to have the uploaded file be put into some kind of workflow-queue for further processing. What is the best way and where to begin in the code?

The scenario is:

  1. Multiple users could be uploading files at the same time.
  2. The processing of the uploaded file on the server could take any amount of time (during which time the user may even get logged off)
  3. User should see some kind of status on the client side UI that the uploaded file processing is happening (and in case of any errors, the error results).
  4. After the processing is complete, the file-processor might have generated few additional files (to generated reports/charts) to be visible on the front end UI.
  5. If files are arriving at faster pace than can be processed, then they may get piled up at the server (so a persistent file processing queue is required)
  6. If for any reason the server side post-processing of file gets interrupted we should have a mechanism to restart processing the file and other items left in the queue.

Sounds complex, but this is just a typical dashboard functionality of a workflow system. In my research came across the 'power queue' package that sounds reasonably close.

Storing the uploaded files list in MongoDB and tagging each of them with 'processed or not' kind of flag in combination with power queue should achieve this. But being new to JS (coming from C/C++), need some pointers on where to start.

Thanks in advance.

radiegtya commented 9 years ago

Hi Krishna,

According to your question, here are my answer:

  1. of course it is a big yes. I am already using meteoris for my big project (ERP)
  2. Before answering your question, may I know what meteoris version are you using? Because between meteoris v0.9.1 and v0.9.2 there are very big change in code structure. (btw meteoris 0.9.2 now using code generator)

you can watch it in action here: https://www.youtube.com/watch?v=erZvEQkr8kw

Thanks for trying meteoris.

KrishnaPG commented 9 years ago

Thank you - wonderful capability (generating the code on the fly).

Its great to see reflection / introspection in a ready to use package even before it hit Version 1. (Back in c/c++ we had to spend atleast couple of years planning the whole DLL export complexity and even the LLVM is not quite fully there after all these years).

Good job. Keep up the good work.

To answer your question: Sure, I would be shifting to this new version of yours (and if any newer comes, would be adapting to the new ones along the way).

Now that you have mentioned code based on user specified fields, it was originally part of my plan also to attach the 'scehma' of data (specified by user as part of the upload process) along with the uploaded data so that the post-processor reads the schema and processes it based on that. But, I was thinking may be it would take couple of months, if not years.

But after seeing the demo, looks like you are already almost there. Now, the question is: how to use this functionality to let,

  1. user specify the fields of data (along with file upload)
  2. store this specification (schema) along with data in a persistent queue till post-processor is ready
  3. Have a post-processor take up what ever is in the queue waiting and process the data based on its attached schema (as mentioned in the original question).

Pointers would be great help. As indicated earlier, the power queue package could be good starting point I guess. Do you think it can be made to work with meterois?

radiegtya commented 9 years ago

Sorry Mr Krishna, where are you from? Sorry, but I am afraid that my answer will not satisfy you. Because it will be too long to describe it here. And I am not completely sure that my answer are the same with what did you mean.

So let have some talk at facebook or chat messanger.

Can you close this issue then? :)

Thanks

KrishnaPG commented 9 years ago

Not a problem radiegtya. Which ever is convenient to you. I have added you in Google+ circles. Once you are free please ping - we can chat.

Thank you.

andrewmartz-af commented 9 years ago

I have the same question. Have you figured out how to do it?

KrishnaPG commented 9 years ago

Nope - I couldn't catch him for a chat.

Anyhow, since the discussion could be useful to multiple audience now, it would be better to finalize this here (rather than through private chats).

Hope Radiegtya could spare us some time here.

radiegtya commented 9 years ago

Hi Krishna and Drew.

Maybe rather than making upload progress by yourself, You can use some of meteor plugin that already support it. And about updating/inserting progress then logout when process is still incomplete, meteor are already support that kind of thing if You are using Meteor.call method. Just try using simple apps and doing that. Correct Me if I am Wrong. :)

radiegtya commented 9 years ago

I think I'll close this for now. Just reopen if there are another question.