"Cloud Drive" / absolute path mode

[TheDailySpank]

I work with a number of remote machines and have tried other render managers (Pandora Prism, Crowd Render, etc) but found the way they distribute files to be extra upload/download that shouldn't apply since the blend files are already distributed via cloud drives being locally synced to the machines to do the rendering. A lot of the assets for my scenes are linked from other blend files that are all within a sub-folder structure within a "3D" folder.

It would be super helpful if there was an option to use the blend file directly instead of creating new files and syncing them.

The way it would work is you check a checkmark that says "use absolute path" (or similar) so instead of syncing the file, it simply refers to that drive and path.

You specify the render file, say on R:\3D\demo.blend and the client tells the server "It's right here at R:\3D\demo.blend" instead of packing up the files and syncing them since some of the files I use take longer to sync than it does to render a few frames.

Yes, there are caveats, like making sure that all clients have the latest file. Maybe that can be mitigated by a file hash of the file the client is sending to ensure that the remote system has the same version as expected to be rendered. And of course, you would need to have the same drive letter on all platforms so it would break cross-platform capability, but in my case, all machines are running the same OS.

Sorry, I don't know how to label this as an Enhancement... :(

[LogicReinc]

To rephrase your issue and proposition. You want to render the file from an exact location, so that the relative positioning to resource files is retained, meaning you do not need to embed resources? (Since synchronization is faster by the software compared to the cloud, I assume this is your underlying idea)

I do see wider use for this in the scenario of a shared drive, for cloud usage I see it work less efficiently due to the following caviats:

• As you mentioned, need to verify if the file is actually the same
• Cloud synchronization tends to take long as your file is sent to remote servers and may not be synced immediately
• Depending on your usage, you might be working on that file, meaning it changes during rendering, this will cause issues. Thus requiring copying of the file either way (can be done in the same folder)

My proposition would be, allow a checkbox that provides a path to be inserted, which points to an absolute path accessible to all render nodes. Then:

• BlendFarm will "sync" but instead of transfering a file, it will give an absolute path and do the following:
  • Check if the file is the same.
  • Make a copy of the file in the same directory, for example R:/SomeDir/MyBlendFile.blend => R:/SomeDir/MyBlendFile.RenderSync.blend

• BlendFarm will "render", the file it will render is "R:/SomeDir/MyBlendFile.RenderSync.blend" in the given directory

  There are some more issues like how long should BlendFarm wait for a file to change? Will it be blocking or throw an error if it doesn't match? etc etc BlendFarm copies files not just for providing the file, but also to prevent it from being changed. (The master application will also make a copy internally before sending for example)

  Please let me know your view on this and if it makes sense in your scenario.

[TheDailySpank]

Im my scenario, all of the remote computers are on a VPN or on a local network (at the office). All computers with discrete video cards have Blender on them and a mapped drive letter, "R:" to assist with rendering (or are also 3D workstations).

The only time we need lots of rendering power is when rendering out animations. The blend files have a number of linked resources as we re-use the same models over and over, but change placement, different camera moves, etc. These Blend files are relatively small, only a couple of megabytes. Syncing requires all that to be packed which end up making the file a couple of gigs.

The R: drive is actually just a folder synced via NextCloud and SUBST'd to R: or is mapped to a NAS as R: if the computer is at the office where that NAS lives (which itself syncs to NextCloud).

To the end computers, they are all part of the same network and the R: drive is a locally shared network drive.

All of the individual linked blend files are synced in the background over time and very rarely change once built.

Loading a blend with linked resources, via absolute paths, with a slightly different filename (as you proposed) would work since it's only a few megabytes (not gigs). Maybe incorporating the hash (SHA, MD5, etc) of the file in the filename could do double duty of making it verifiable as well as letting the remotes know it's not there yet (different filename than when re-using "RenderSync" every time) and to wait a certain amount of time for it to become available through the cloud sync mechanism once the server sends the render job to the client. Probably should be user configurable as some remote nodes have slower download than others.

For example, the classroom.zip demo file's MD5 is 3adbb7114b514bfc6fc724ce20f86b4e so the filename could be R:/SomeDir/classroom.3adbb7114b514bfc6fc724ce20f86b4e.blend. Make a change and the MD5 will change and therefore a new file will be created and not be confused with the previous version.

Changing the underlying linked files could cause one render node to render something different than the others, but that's inherent to this style of workflow so nothing really changes.

If the file doesn't match the hash in the filename, it should throw an error since that file is corrupt, no matter who's looking at it.

[LogicReinc]

In your scenario syncing a copy of the file and writing it to the same location is likely still faster than waiting for NextCloud to sync seeing how the file is small and is send within a second, and it solves a lot of other issues.

Since you are using it to render animations, do keep in mind BlendFarm's animation rendering is still a bit a work in progress and performance doesn't scale perfectly depending on your Blendfile (see Animation Frame Batching issue).

Either way what you describe is roughly what I had in mind. I'll give this issue some priority but will likely be next month as I'm quite busy the rest of January. Hopefully I can implement this in February, seeing how this also resolves resource resolution without embedding it in some situation, it could be very useful (mostly wasn't implemented because its not as dummy proof and requires additional setup).

[LogicReinc]

A minor side note, in case you werent aware, you mentioned all computers have Blender on them. Do know that BlendFarm uses its own versions of Blender, and doesn't use any existing Blender installation (it keeps specific versions in its Blender_Data directory)

[TheDailySpank]

Thank's for looking into adding this option. I think it will help at least a few people out.

I saw that it will download as necessary and was thinking maybe it too could be shared via the absolute path method. This way we only end up downloading it from blender.org once to keep their bandwidth usage down too.

Each machine has a local copy because at the moment, I am remotely accessing the machines and specifying frame ranges as that's faster than waiting on a large sync to complete with the existing render solutions.

[LogicReinc]

v1.0.9 adds a new feature that likely solves this issue. Under the "System" tab, it allows you to check "Use Networked Path". It will show you 3 text fields labeled Windows/Linux/MacOS. The fields of the platforms you use should be filled with an absolute path on which the current blend file is accessable. (eg. On Windows you have a blendfile in a shared directory "C://Shared/MyBlendFile.blend", your computer has the Shared directory shared. The computer IP is 192.168.1.123, then you'd enter something like //192.168.1.123/Shared/MyBlendFile.blend for windows. And on Linux you might have that share mounted under /mnt/Shared etc).

This should allow you to

• Use non-embedded resources relative to the network path.
• Effectively skip syncing (as this sync will only check if the file exists on the render node, not transfer anything)

[LogicReinc]

If you have no further notes, I'll close this issue in a few days