squid-box / SevenZipSharp

Fork of SevenZipSharp on CodePlex
GNU Lesser General Public License v3.0
274 stars 101 forks source link

Volume parts are exclusively in-use until compression fully completed #177

Open fatihsoydan opened 1 year ago

fatihsoydan commented 1 year ago

Hello, I am trying to write a project that uploads big folder. I am compressing to volumes. When a volume file completed, I want to start upload. I guess, but It is exclusively opened for writing until the compression fully completed. So when I want to upload last completed volume, windows couldn't access the volume file.

Could we close the finished parts ?

Here is my code :

internal void  GenerateZip(string fileOrfolder, string zipFileName) {
            zipReady = false;
            SevenZipBase.SetLibraryPath(AppDomain.CurrentDomain.BaseDirectory+"7z.dll");
            SevenZipCompressor zip = new SevenZipCompressor();  
            zip.PreserveDirectoryRoot = false;
            zip.VolumeSize = 100 * 1024 * 1024;
            zip.Compressing += Zip_Compressing;
            zip.ArchiveFormat = OutArchiveFormat.SevenZip;
            zip.CompressionMode = CompressionMode.Create;

            FileAttributes fa = File.GetAttributes(fileOrfolder);
            if (fa.HasFlag(FileAttributes.Directory)) {
                zip.CompressDirectory(fileOrfolder, zipFileName);
            }  else {
                zip.CompressFiles(zipFileName, fileOrfolder);
            }
            zipReady = true;
        }

internal void UploadFinished(string zipFileName) {
            currentPart = 1;
            string folder = Path.GetDirectoryName(zipFileName); 
            while(!zipReady) {
                foreach (var file in Directory.GetFiles(folder)) {
            // When Sevenzip works on part3, I want to upload part1
                    string beklenen = Path.Combine(folder , this.uuid + ".zip." + (currentPart+2).ToString("D3"));
                    if (file==beklenen) {
                        string uploadfile = Path.Combine(folder, this.uuid + ".zip." + currentPart.ToString("D3"));
                        Console.WriteLine("Upload Et [" + currentPart.ToString() + "] " + uploadfile); ;
                        apiManager.UploadBackup(BackupID,currentPart,uploadfile); 
                        Thread.Sleep(5000);
                        currentPart++;
                    } 
                }
                Thread.Sleep(2000);
            }
            // Upload last 2 part  here when compression finished
        }

public void PrepareAndSendResult(string fileOrfolder, string zipFileName) {
    var t1 = Task.Run(() => GenerateZip(fileOrfolder, zipFileName));
    var t2 = Task.Run(() => UploadFinished(zipFileName));
    Task.WaitAll(new[] { t1, t2 }); 
    Console.WriteLine("==== ALL DONE ===");
}
fatihsoydan commented 1 year ago

I found a comment about my issue :

1) 7-zip writes 7z header at start of archive (at first volume) at the end of operation. 2) The code that creates volumes doesn't know what exact seek and write operations are required for 7z code. So it keeps all volumes open. For example, zip code requires another seek and write operations (writing to any volume). No way to fix it now. Maybe there is way to ignore open-lock with some special program. In that case you can try to remove any volume except of volume .7z.001.

fatihsoydan commented 1 year ago

I wrote a project with Golang describing what I need.
https://github.com/fatihsoydan/gzip2parts

I will use this project as process for solving my problem (just temporary).
But I think, there are a lot of people working with big files. If we can add this future in SevenZipSharp repository that will be fantastic.