Request - Send the file with the job

[Juang3d]

Hi there.

The idea is what is said in the title, whenever you send a job to the farm, send also the file to the server so it can be transferred to the renderers to a temporary file.

This means that you won't be rendering over the final file, but over a temporary file, also this means that you don't have to put the file in a shared folder for all the farm, of course you will have to have the textures and the rest of dependencies, but not the file itself (in my case the .max file)

This is useful in many cases, in the case you have to modify your scene after you send it to render, or for example if you have to repeat a render with exactly the same scene, you will be able to repeat it from the server, no need to re-send the file or the job, because it will be temporarily stored with the server, and if you modified yor scene further, you won't have to worry to have the wrong scene rendering.

Also this is useful in my specific case (that may be the case of others too)

In my case I have my office in a different place than my farm, they are connected via internet, so I don't have a shared folder with the scenes, whenever I sedn a job it should be sent with the file, this is automatically done by backburner right now, but I would like to abandon backburner and use CGRU.

Since CGRU don't send the file automatically this includes a new human-error possibility that I would like to avoid, so having the client sending the temporary file to the server automatically with the job could be great and will solve a lot of possible problems.

In the end, what I would like to request is that CGRU transfers the job file automatically to the farm and put it in a temporary folder in the server, and when a renderer starts the job it grabs the job file form the server, puts it in a temporary folder and starts working with it, any other dependencies should be in a shared folder or a local folder and have the same route (for example x:\work\textures\") but the file itself will be transferred to the renderers automatically.

Hope you like this request, I think it is pretty useful in general.

[Juang3d]

Is this not interesting to anyone or it's just that this is already implemented and I did not notice?

Because this has been openend a bunch of times but no one seems to respond to this.

Cheers!

[timurhai]

Hi. ( i missed your first question email notification, sorry ) Afanasy (submission scripts) does not do it "out-of-the-box". But it is interesting. And it can be a useful feature.

But we store any project files on a fileserver only. If somebody needs farm form home (for example), he uses VPN or some teamviewer, and stores all files on a file server too. So we do not need such feature. But the project is open-source and you(somebody) can write anything. I can just guide and help.

PS Better to use some VPN or teamviewer - much more control. Submission script can miss something, this way is much complex (job submission) than a local network rendering, you should take care of paths mapping and temporary folders.

[Juang3d]

No problem Timur.

I was talking about sending just the scene file, I think this is useful because usually textures and dependencies area easy to handle and are less prone to changes, but the scene can have a lot of variations, you can have textures and similar things in a network share, but the scene is usually in the local computer (or handled by some kind of asset manager) so if you send a render you will need all the renderers with the same network share, the problem is that this network share is not the same as textures, because each artist may have it's own scene folder, even inside of an organized folder structure inside a studio, so replicating this structure in a network share may be problematic, so if the server manages the scene file, so it just sends the scene file to the renderers, put this file (.max, .blend, .ma...) in a temporary folder, the scnee can be open by any software, and the textures will still be mapped to that network share, but you don't need to synch your scene working folder with all the renderers, so any artist can send a file to render without worrying if their scee is in sync or not.

I'm not sure if I'm explaining very well, but in my personal case, this is important, not just if I sedn renders from home, but also in the case that we are two persons working with the farm, we can't have the scenes folder of both artists configured int he farm, but if the farm manages the scene file, we can forget about it, even when we need to have the textures folder and the render output folder in the same place, but it's not our work place, plus if you need to re-render the exact same scene you just have to re-run the job, so you don't have to worry about what was the rendered file or version because it's stored in the temporary farm folder, and it remains there until you delete the job, it's like having a snapshot of the rendered scene, if you have to repeat something you are sure that everything will remain exactly the same way it was.

I'm not sure how to develop this, my programming skills are a bit low, I just do some ios, android and maxscript programming, I could be happy to implement this, but I think I'm not able to do that.

Hope you like the feature.

Cheers!

2015-03-30 21:00 GMT+02:00 Timur Hairulin notifications@github.com:

Hi. ( i missed your first question email notification, sorry ) Afanasy (submission scripts) does not do it "out-of-the-box". But it is interesting. And it can be a useful feature.

But we store any project files on a fileserver only. If somebody needs farm form home (for example), he uses VPN or some teamviewer, and stores all files on a file server too. So we do not need such feature. But the project is open-source and you(somebody) can write anything. I can just guide and help.

PS Better to use some VPN or teamviewer - much more control. Submission script can miss something, this way is much complex (job submission) than a local network rendering, you should take care of paths mapping and temporary folders.

— Reply to this email directly or view it on GitHub https://github.com/CGRU/cgru/issues/126#issuecomment-87791768.

[timurhai]

Almost all afanasy submission scripts creates a temp scene file to render (copy current with a prefix), so artist can continue to change scene (a and save it) and it will not affect render sequence.

I will not write it, i have no time and lots of other tasks, i can just guide.

[ultra-sonic]

i personally don't like that idea of sending scenefiles through afanasy, but if necessary you could easily script that "pre-sync" on your own. I am doing this right now with a project for cloud rendering on amazon. You can check it out here but it is not in a state where you can easily use it. I am using afanasy in that project as well... take a look here: https://github.com/ultra-sonic/opencloudrender

cheers Oli

On Tue, 31 Mar 2015 at 12:19 Timur Hairulin notifications@github.com wrote:

Almost all afanasy submission scripts creates a temp scene file to render (copy current with a prefix), so artist can continue to change scene (a and save it) and it will not affect render sequence.

I will not write it, i have no time and lots of other tasks, i can just guide.

— Reply to this email directly or view it on GitHub https://github.com/CGRU/cgru/issues/126#issuecomment-88030105.

[Juang3d]

That is exactly the situation I'm talking about, but instead of having a series of Amazon machines, I have my own machines as my own private cloud computing system, anyways, I think it could be easier to do what you say, create my own pre-sync tool. I'll look into your project for sure, thanks :)

Cheers!

2015-04-04 15:12 GMT+02:00 Oliver Markowski notifications@github.com:

i personally don't like that idea of sending scenefiles through afanasy, but if necessary you could easily script that "pre-sync" on your own. I am doing this right now with a project for cloud rendering on amazon. You can check it out here but it is not in a state where you can easily use it. I am using afanasy in that project as well... take a look here: https://github.com/ultra-sonic/opencloudrender

cheers Oli

On Tue, 31 Mar 2015 at 12:19 Timur Hairulin notifications@github.com wrote:

Almost all afanasy submission scripts creates a temp scene file to render (copy current with a prefix), so artist can continue to change scene (a and save it) and it will not affect render sequence.

I will not write it, i have no time and lots of other tasks, i can just guide.

— Reply to this email directly or view it on GitHub https://github.com/CGRU/cgru/issues/126#issuecomment-88030105.

— Reply to this email directly or view it on GitHub https://github.com/CGRU/cgru/issues/126#issuecomment-89575096.