Force delete untracked files when doing downsync?

[AOOSG]

When using longtail to download new UE5 versions: It looks like Unreal Engine generates some new files in the downsync folder while compiling / running the game.

This seems to cause problems when using downsync to download the latest UE5 version (5.2 -> 5.3 update this time) where some folders cannot be deleted. Here's the error:

ERRO Can't to remove dir `D:/rapid-tests/UE5/Engine/Plugins/Experimental/NNE/Intermediate/Build/Win64/x64/UnrealEditor/Development/NNEORT`, failed with 41  block_store_api=000001b9adfe8080 file="D:\\a\\longtail\\longtail\\src\\longtail.c" func=Longtail_ChangeVersion hash_api=000001b9878eeee0 job_api=000001b9878ede90 line=7642 optional_cancel_api=0000000000000000 optional_cancel_token=0000000000000000 progress_api=000001b9ae0c7890 retain_permissions=1 source_version=000001b9b3303050 store_index=000001b9bb211050 target_version=000001b9aeb19050 version_diff=000001b9adfe81c0 version_path="D:/rapid-tests/UE5" version_storage_api=000001b9878eeda0

Looking in D:/rapid-tests/UE5/Engine/Plugins/Experimental/NNE/Intermediate/Build/Win64/x64/UnrealEditor/Development/NNEORT there's a file:

PCH.NNEOnnxruntime.h

Is there a way to just force delete whatever files is preventing folders from being deleted and a downsync to complete successfully? It should be safe to delete any files not expected inside the D:/rapid-tests/UE5 folder.

[DanEngelbrecht]

This could likley be resolved by having golongtail ignore certain folders.

The quick fix is to remove the .longtail.index.cache.lvi in the root of the target folder.

To avoid future issue I think you should add a regex to exclude certain folders from the upsync - you are uploading intermediate build data as part of the upload, I don't think that is intentional?

Something like --exclude-filter-regex .*\/Intermediate\/.* should probably work.

[AOOSG]

Ah good suggestion, will try the exclude regex.

So the reason for the problem is that more files (untracked files are added when compiling/running the game) were added to a folder, so it can't delete the folder during a downsync.

In this case the folder is a subfolder of an Intermediate folder anyways, so files there shouldn't need to be tracked in the first place.

[DanEngelbrecht]

Correct, just make sure you use the exclude expression when you upload the version, it is not useful in this scenario when downloading.

[DanEngelbrecht]

You might want to fix the old version as that probably contains the same type of bad data.

[AOOSG]

Unfortunately it looks like at least one file in an Intermediatefolder is required, I can't generate the project files unless BuildRules.dll is found (and it's inside an Intermediate folder).

What does removing .longtail.index.cache.lvi do? Completely cleans the folder upon the next downsync? That sounds good as well.

[DanEngelbrecht]

Yes, the file keeps an index of what is currently on disk. You can use —no—cache-target-index with downsync to prevent it from being created.

[AOOSG]

Thanks, I think I'll go with that flag, I just need to double check that the build times aren't increased by a lot due to any intermediate files being deleted in the UE5 folder between each build.

It also seems that running downsync the first time (when the error occurs) automatically deletes .longtail.index.cache.lvi so this is also "fixed" by running downsync twice.

[AOOSG]

I ended up using —no—cache-target-index and all is well now! Thanks.