Licarious / MNR_AutoIncrement

Auto Increment for CK2
0 stars 0 forks source link

IRIG106 findings #1

Open Licarious opened 10 months ago

Licarious commented 10 months ago

https://www.ddc-web.com/resources/FileManager/dbi/Whitepapers/IRIG106.pdf pg 5

Relative time is determined by the output of a free-running 48-bit 10 MHz binary counter. If relative time is used, it is distributed to all channels within the same data recorder, thereby providing a local common time base for all stored data, regardless of the data type. When relative rather than absolute time is used, the standard requires that it be accurate to within ±100 µS of absolute time, with a recommended accuracy of µS.

https://apps.dtic.mil/sti/trecms/pdf/AD1183569.pdf pg 38

The PCM Data Packet body contains in case of Throughput Mode (indicated in the header) the not synchronized and not aligned bit stream of the PCM data. The Relative Time Counter value corresponds to the first data bit in the packet, which can be any bit in the major frame.

Can multiple packets be sent at the same time? Are the packet recorded to the Chapter 10 file at the start of the bit stream or at the end?

pg 39

The reference of the Intra-Packet Time Stamp is indicated in the Channel Specific Data of the packet with the Time Tag Bits (TTB): 00 = Last bit of the last word of the message 01 = First bit of the first word of the message 10 = Last bit of the first word (command word) of the message

http://ftp.irig106.org/wiki/ch10_handbook:data_file_interpretation

Data packets are written to disk roughly in the time order in which they are received, but data packets and data messages can occur in the data file out of time order. This order can occur because data recorders receive data simultaneously on multiple channels, each channel buffering data for a period of time and then writing it to disk. Therefore, individual data messages will, in general, be somewhat out of time order because of grouping by channel. Consider the case of two 1553 channels recording the same bus at the same time in an identical fashion. Each channel receives, buffers, and writes data to disk. The first channel will write its buffered data to disk followed by the second channel. The data received from the second channel will be from the same time period as the data from the first channel and will have identical time stamps but will be recorded after the first channel in the data file.

Starting with the IRIG 106-05 standard, recorders are only allowed to buffer data for a maximum of 100 milliseconds and data packets must be written to disk within one second. This ensures that data packets can only be out of time order by a maximum of one second. Be warned, though, that the maximum amount of time data packets can be out of order for data files produced before IRIG 106-05 is unbounded and it is not unusual to encounter data files with data packets five or more seconds out of time order

Licarious commented 8 months ago
public static void main(String[] args) throws IOException {
      String zipPath = "C:\\Users\\fligh\\OneDrive\\Desktop\\Java IRMap Output\\ChannelData.zip";
      File `readFile` = new File("C:\\Users\\fligh\\OneDrive\\Desktop\\Java IRMap Output\\ChannelData.txt");

      // Create a text file with "hello world" in it
      File file = new File("Test.txt");
      try {
          FileWriter fileWriter = new FileWriter(file);
          fileWriter.write("hello world2");
          fileWriter.close();
      } catch (Exception e) {
          e.printStackTrace();
      }

      // Add files to the zip file
      addFilesToZip(Arrays.asList(file), zipPath);
      addFilesToZip(Arrays.asList(readFile), zipPath);
  }
public static void addFilesToZip(List<File> files, String zipPath) throws IOException {
      try (FileSystem fs = FileSystems.newFileSystem(Paths.get(zipPath), (ClassLoader) null)) {
          Path root = fs.getPath("/");

          // For each file, copy it into the zip file
          for (File file : files) {
           System.out.println("File Added: " + file.getName());
              // Check if file with same name already exists in the zip
              Path zipEntryPath = root.resolve(file.getName());
              if (Files.exists(zipEntryPath)) {
                  // Delete the existing file
                  Files.delete(zipEntryPath);
              }

              // Add file to the zip
              try (InputStream is = new FileInputStream(file);
                   OutputStream os = Files.newOutputStream(zipEntryPath)) {
                  byte[] buffer = new byte[1024];
                  int bytesRead;
                  while ((bytesRead = is.read(buffer)) != -1) {
                      os.write(buffer, 0, bytesRead);
                  }
              }
          }
      }
  }
Licarious commented 8 months ago
public record FilePathRecord(Path outerZip, Path innerZip, Path file) {}
public record FilePathRecordOptional(Optional<Path> outerZip, Optional<Path> innerZip, Path file) {}
public static void writeToNestedZip(FilePathRecord record, byte[] fileContent) throws IOException {
    // Check if the outer zip file exists, if not, create it
    if (!Files.exists(record.outerZip)) {
        createNestedZip(record.outerZip);
    }

    try (FileSystem fs = FileSystems.newFileSystem(record.outerZip, (ClassLoader) null)) {
        // Check if the inner zip file exists, if not, create it
        if (!Files.exists(fs.getPath(record.innerZip.toString()))) {
            createNestedZip(record.innerZip);
        }

        // Write file content to inner zip file
        writeToZip(fs.getPath(record.innerZip.toString()), record.file.getFileName().toString(), fileContent);
    }
}
private static void createNestedZip(Path zipFilePath) throws IOException {
    try (ZipOutputStream outerZipOutputStream = new ZipOutputStream(Files.newOutputStream(zipFilePath))) {}
}
private static void writeToZip(Path zipFilePath, String fileName, byte[] content) throws IOException {
    // Create the zip file if it doesn't exist
    if (!Files.exists(zipFilePath)) {
        createNestedZip(zipFilePath);
    }

    // Open the zip file system and write the content
    try (FileSystem fs = FileSystems.newFileSystem(zipFilePath, (ClassLoader) null)) {
        Path file = fs.getPath(fileName);
        Files.write(file, content);
    }
}
public static void writeToNestedZip(FilePathRecordOptional record, byte[] fileContent) throws IOException {
    if(record.outerZip.isPresent() && record.innerZip.isPresent()) {
        // Check if the outer zip file exists, if not, create it
            if (!Files.exists(record.outerZip.get())) {
                createNestedZip(record.outerZip.get());
            }

            try (FileSystem fs = FileSystems.newFileSystem(record.outerZip.get(), (ClassLoader) null)) {
                // Check if the inner zip file exists, if not, create it
                if (!Files.exists(fs.getPath(record.innerZip.toString()))) {
                    createNestedZip(record.innerZip.get());
                }

                // Write file content to inner zip file
                writeToZip(fs.getPath(record.innerZip.get().toString()), record.file.getFileName().toString(), fileContent);
            }
        }
        else if(record.outerZip.isPresent()) {
        // Check if the outer zip file exists, if not, create it
            if (!Files.exists(record.outerZip.get())) {
                createNestedZip(record.outerZip.get());
            }

            // Write file content to outer zip file
            writeToZip(record.outerZip.get(), record.file.getFileName().toString(), fileContent);
    }
    else if(record.innerZip.isPresent()) {
        // Check if the inner zip file exists, if not, create it
        if (!Files.exists(record.innerZip.get())) {
            createNestedZip(record.innerZip.get());
        }

        // Write file content to inner zip file
        writeToZip(record.innerZip.get(), record.file.getFileName().toString(), fileContent);
    } 
    //I don't think we will need this last else statement
    else {
        // Write file content to file
        Files.write(record.file, fileContent);
    }
}
Licarious commented 6 months ago
public record zipedFileRecord(Path innerZip, Path file, String parrentFolder) {}
public static List<zipedFileRecord> getAllFilesWithNameInZip(Path zipFilePath, String fileName) throws IOException {
    List<zipedFileRecord> fileList = new ArrayList<>();

    //if one of the files is a zip file, we need to open it
    try (FileSystem fs = FileSystems.newFileSystem(zipFilePath, (ClassLoader) null)) {
        Files.walk(fs.getPath("/")).filter(Files::isRegularFile)
            //if the file is a zip file we need to open it by checking the first 4 bytes of the file
            .filter(path -> {
                try {
                    return Files.readAllBytes(path).length >
                    4 &&
                    Files.readAllBytes(path)[0] == 0x50 &&
                    Files.readAllBytes(path)[1] == 0x4B &&
                    Files.readAllBytes(path)[2] == 0x03 &&
                    Files.readAllBytes(path)[3] == 0x04;
                    } catch (IOException e) {
                        return false;
                }
            })
            .forEach(path -> {
                try {
                    //if the file is a zip file, we need to open it
                    try (FileSystem fs2 = FileSystems.newFileSystem
                            (path, (ClassLoader) null)) {
                        Files.walk(fs2.getPath("/")).filter(Files::isRegularFile)
                            .filter(path2 -> path2.getFileName().toString().equals(fileName))
                            //add the file to the list using path as the inner zip and path2 as the file, and just the name of the parrent folder as the parrent folder
                            .forEach(path2 -> fileList.add(new zipedFileRecord(path, path2, path2.getParent().getFileName().toString())));
                            //.forEach(path2 -> fileList.add(new zipedFileRecord(path, path2, path2.getParent().toString())));
                    }
                    } catch (IOException e) {
                        // Handle IOException
                        e.printStackTrace();
                    }
            });

    }

    //print out the files
    System.out.println("Files with name: " + fileName + " in zip: " + zipFilePath.toString());
    for (zipedFileRecord fileRecord : fileList) {
        System.out.println("Inner Zip: " + fileRecord.innerZip.toString() + " File: " + fileRecord.file.toString()
                + " Parrent Folder: " + fileRecord.parrentFolder);
    }

    return fileList;
}
Licarious commented 6 months ago

`/**

Licarious commented 6 months ago
public static void importZipFile(Path zipPath, Path folder, Path importPath) throws IOException {
    if (!Files.exists(zipPath)) {
        //create the zip file if it doesn't exist
        createZip(zipPath);
    }

    try (FileSystem fs = FileSystems.newFileSystem(zipPath, (ClassLoader) null)) {
        try (ZipInputStream zis = new ZipInputStream(Files.newInputStream(importPath))) {
            ZipEntry zipEntry;
            while ((zipEntry = zis.getNextEntry()) != null) {
                Path filePath = fs.getPath(folder.toString(), zipEntry.getName());
                Files.createDirectories(filePath.getParent());
                Files.copy(zis, filePath, StandardCopyOption.REPLACE_EXISTING);
            }
        }
    }
}

public static void importZipFile2(Path zipPath, Path folder, Path importPath) throws IOException {
    if (!Files.exists(zipPath)) {
        // Create the zip file if it doesn't exist
        createZip(zipPath);
    }

 // Open the zip file system
    try (FileSystem zipFs = FileSystems.newFileSystem(zipPath, (ClassLoader) null)) {
        // Open the import zip file
        try (ZipFile zipFile = new ZipFile(importPath.toFile())) {
            Enumeration<? extends ZipEntry> entries = zipFile.entries();
            while (entries.hasMoreElements()) {
                ZipEntry entry = entries.nextElement();
                Path filePath = zipFs.getPath(folder.toString(), entry.getName());

                if (entry.isDirectory()) {
                    Files.createDirectories(filePath);
                } else {
                    Files.createDirectories(filePath.getParent());
                    try (InputStream is = zipFile.getInputStream(entry)) {
                        Files.copy(is, filePath, StandardCopyOption.REPLACE_EXISTING);
                    }
                }
            }
        }
    }
}

public static void importZipFile3(Path zipPath, Path folder, Path importPath) throws IOException { //this one causes problems for the other 2
    if (!Files.exists(zipPath)) {
        // Create the zip file if it doesn't exist
        createZip(zipPath);
    }

    try (ZipOutputStream zos = new ZipOutputStream(Files.newOutputStream(zipPath, StandardOpenOption.CREATE, StandardOpenOption.APPEND));
         ZipFile zipFile = new ZipFile(importPath.toFile())) {

        Enumeration<? extends ZipEntry> entries = zipFile.entries();
        while (entries.hasMoreElements()) {
            ZipEntry entry = entries.nextElement();
            String entryName = folder.resolve(entry.getName()).toString().replace(File.separatorChar, '/');
            ZipEntry newEntry = new ZipEntry(entryName);

            if (entry.isDirectory()) {
                zos.putNextEntry(newEntry);
                zos.closeEntry();
            } else {
                zos.putNextEntry(newEntry);
                try (InputStream is = zipFile.getInputStream(entry)) {
                    byte[] buffer = new byte[1024];
                    int len;
                    while ((len = is.read(buffer)) > 0) {
                        zos.write(buffer, 0, len);
                    }
                }
                System.out.println("Imported file: " + entryName);
                zos.closeEntry();
            }
        }
    }
}
Licarious commented 6 months ago


import static org.junit.jupiter.api.Assertions.assertArrayEquals;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertTrue;

import java.io.IOException;
import java.nio.file.FileSystem;
import java.nio.file.FileSystems;
import java.nio.file.FileVisitResult;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.nio.file.SimpleFileVisitor;
import java.nio.file.attribute.BasicFileAttributes;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;

import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;

public class ZipHandlerTest {

    private Path tempZip;
    private Path tempDir;
    private static final byte[] SAMPLE_CONTENT = "Sample Content".getBytes();

    @BeforeEach
    public void setUp() throws IOException {
        tempDir = Paths.get("testFolder");
        if (!Files.exists(tempDir)) {
            Files.createDirectory(tempDir);
        }
        tempZip = tempDir.resolve("test.zip");
    }

    @AfterEach
    public void tearDown() throws IOException {
        Files.walkFileTree(tempDir, new SimpleFileVisitor<Path>() {
            @Override
            public FileVisitResult visitFile(Path file, BasicFileAttributes attrs) throws IOException {
                //if a file is deleted, print its name
                Files.delete(file);
                return FileVisitResult.CONTINUE;
            }

            @Override
            public FileVisitResult postVisitDirectory(Path dir, IOException exc) throws IOException {
                //if a directory is deleted, print its name
                Files.delete(dir);
                return FileVisitResult.CONTINUE;
            }
        });
    }

    @Test
    public void testWriteToZipCreatesZipFileWhenZipPathDoesNotExist() throws IOException {
        Path filePath = Path.of("folder/file.txt");
        ZipHandler.writeToZip(tempZip, filePath, SAMPLE_CONTENT);
        assertTrue(Files.exists(tempZip));
    }

    @Test
    public void testWriteToZipCreatesParentDirectoriesWhenParentDirectoryIsNull() throws IOException {
        Path filePath = Path.of("file.txt"); // No parent directory
        ZipHandler.writeToZip(tempZip, filePath, SAMPLE_CONTENT);

        // Verify that the file exists in the zip
        try (FileSystem fs = FileSystems.newFileSystem(tempZip, (ClassLoader) null)) {
            Path fileInZip = fs.getPath(filePath.toString());
            assertTrue(Files.exists(fileInZip));
        }
    }

    @Test
    public void testWriteToZipDoesNotRecreateExistingParentDirectories() throws IOException {
        // Create a parent directory before calling the writeToZip method
        Path parentDir = Files.createTempDirectory("parentDir");
        Path filePath = parentDir.resolve("file.txt");

        // Write to the zip with an existing parent directory
        ZipHandler.writeToZip(tempZip, filePath, SAMPLE_CONTENT);

        // Verify that the parent directory exists and only created once
        try (FileSystem fs = FileSystems.newFileSystem(tempZip, (ClassLoader) null)) {
            Path parentInZip = fs.getPath(parentDir.toString());
            assertTrue(Files.exists(parentInZip));
            assertEquals(1, Files.list(parentInZip).count()); // Only one file should exist
        }
    }

    @Test
    public void testWriteToZipDoesNotOverwriteExistingFile() throws IOException {
        // Write a file with the same name to the zip
        Path existingFilePath = Path.of("folder/file.txt");
        ZipHandler.writeToZip(tempZip, existingFilePath, SAMPLE_CONTENT);

        // Write the same file to the zip again
        ZipHandler.writeToZip(tempZip, existingFilePath, SAMPLE_CONTENT);

        // Verify that only one file exists in the zip
        try (FileSystem fs = FileSystems.newFileSystem(tempZip, (ClassLoader) null)) {
            Path fileInZip = fs.getPath(existingFilePath.toString());
            assertTrue(Files.exists(fileInZip));
            assertEquals(1, Files.list(fileInZip.getParent()).count()); // Only one file should exist
        }
    }

    @Test
    public void testWriteToZipWritesContent() throws IOException {
        Path filePath = Paths.get("folder/file.txt");
        ZipHandler.writeToZip(tempZip, filePath, SAMPLE_CONTENT);

        try (FileSystem fs = FileSystems.newFileSystem(tempZip, (ClassLoader) null)) {
            Path fileInZip = fs.getPath(filePath.toString());
            assertTrue(Files.exists(fileInZip));
            assertArrayEquals(SAMPLE_CONTENT, Files.readAllBytes(fileInZip));
        }
    }

    @Test
    public void testWriteToZipCreatesDirectories() throws IOException {
        Path filePath = Paths.get("folder/subfolder/file.txt");
        ZipHandler.writeToZip(tempZip, filePath, SAMPLE_CONTENT);

        try (FileSystem fs = FileSystems.newFileSystem(tempZip, (ClassLoader) null)) {
            Path dirInZip = fs.getPath("folder/subfolder");
            assertTrue(Files.exists(dirInZip));
            assertTrue(Files.isDirectory(dirInZip));
        }
    }

    @Test
    public void testDeleteAllSubfoldersExcept() throws IOException {
        // Create multiple folders and files
        Path validFolder = Paths.get("folder/valid");
        Path invalidFolder = Paths.get("folder/invalid");
        Path validFile = validFolder.resolve("file.txt");
        Path invalidFile = invalidFolder.resolve("file.txt");
        ZipHandler.writeToZip(tempZip, validFile, SAMPLE_CONTENT);
        ZipHandler.writeToZip(tempZip, invalidFile, SAMPLE_CONTENT);

        // Delete all subfolders except the valid one
        List<Path> validPaths = Collections.singletonList(validFolder);
        ZipHandler.deleteAllSubfoldersExcept(tempZip, Paths.get("folder"), validPaths);

        // Verify valid folder exists and invalid folder is deleted
        try (FileSystem fs = FileSystems.newFileSystem(tempZip, (ClassLoader) null)) {
            assertTrue(Files.exists(fs.getPath(validFolder.toString())));
            assertFalse(Files.exists(fs.getPath(invalidFolder.toString())));
        }
    }

    @Test
    public void testExportZipFile() throws IOException {
        // Write a file to the zip
        Path folderPath = Paths.get("folder");
        Path filePath = folderPath.resolve("file.txt");
        ZipHandler.writeToZip(tempZip, filePath, SAMPLE_CONTENT);

        // List contents of the original zip
        System.out.println("Original zip contents:");

        // Export the contents of the zip to a new zip
        Path exportZip = tempDir.resolve("export.zip");
        System.out.println("Exporting zip from: " + tempZip + " to: " + exportZip);
        ZipHandler.exportZipFile(tempZip, folderPath, exportZip);

        // List contents of the exported zip
        System.out.println("Exported zip contents:");

        // Verify the new zip has the file directly under the root
        try (FileSystem fs = FileSystems.newFileSystem(exportZip, (ClassLoader) null)) {
            Path fileInExportZip = fs.getPath("file.txt");
            System.out.println("Checking existence of: " + fileInExportZip);
            assertTrue(Files.exists(fileInExportZip), "File should exist in the exported zip");
            assertArrayEquals(SAMPLE_CONTENT, Files.readAllBytes(fileInExportZip), "File content should match");
        }
    }

    @Test
    public void testImportZipFile() throws IOException {
        // Create a zip file to import from
        Path importZip = tempDir.resolve("import.zip");
        ZipHandler.writeToZip(importZip, Paths.get("importFolder", "file.txt"), SAMPLE_CONTENT);

        // Import the contents of the import zip into the original zip
        Path folderPath = Paths.get("importFolder");
        ZipHandler.importZipFile(tempZip, folderPath, importZip);

        // List contents of the original zip after import
        System.out.println("Contents of original zip after import:");

        // Verify the original zip has the imported file directly under the specified folder
        try (FileSystem fs = FileSystems.newFileSystem(tempZip, (ClassLoader) null)) {
            // Adjusted path to check for the file directly under the importFolder
            Path fileInZip = fs.getPath(folderPath.toString(), "importFolder", "file.txt");
            System.out.println("Checking existence of: " + fileInZip);
            assertTrue(Files.exists(fileInZip), "File should exist in the original zip under the specified folder");
            assertArrayEquals(SAMPLE_CONTENT, Files.readAllBytes(fileInZip), "File content should match");
        }

        // also a test when the tmpZip file does not exist
        Path nonExistentZip = tempDir.resolve("nonexistent.zip");
        ZipHandler.importZipFile(nonExistentZip, folderPath, importZip);
        try (FileSystem fs = FileSystems.newFileSystem(nonExistentZip, (ClassLoader) null)) {
            // Adjusted path to check for the file directly under the importFolder
            Path fileInZip = fs.getPath(folderPath.toString(), "importFolder", "file.txt");
            System.out.println("Checking existence of: " + fileInZip);
            assertTrue(Files.exists(fileInZip), "File should exist in the original zip under the specified folder");
            assertArrayEquals(SAMPLE_CONTENT, Files.readAllBytes(fileInZip), "File content should match");
        }

    }

    @Test
    public void testImportZipFileWhenZipDoesNotExist() throws IOException {
        // Create a zip file to import from
        Path importZip = tempDir.resolve("import.zip");
        ZipHandler.writeToZip(importZip, Paths.get("importFolder", "file.txt"), SAMPLE_CONTENT);

        // Import the contents of the import zip into the original zip
        Path folderPath = Paths.get("importFolder");

        // also a test when the tmpZip file does not exist
        Path nonExistentZip = tempDir.resolve("nonexistent.zip");

        //check if nonExistentZip exists
        assertFalse(Files.exists(nonExistentZip), "The non-existent zip file should not exist");

        ZipHandler.importZipFile(nonExistentZip, folderPath, importZip);
        try (FileSystem fs = FileSystems.newFileSystem(nonExistentZip, (ClassLoader) null)) {
            // Adjusted path to check for the file directly under the importFolder
            Path fileInZip = fs.getPath(folderPath.toString(), "importFolder", "file.txt");
            System.out.println("Checking existence of: " + fileInZip);
            assertTrue(Files.exists(fileInZip), "File should exist in the original zip under the specified folder");
            assertArrayEquals(SAMPLE_CONTENT, Files.readAllBytes(fileInZip), "File content should match");
        }

    }

    @Test
    public void testImportZipFileDoesNotCreateZipFileIfAlreadyExists() throws IOException {
        // Create a zip file
        Path filePath = Path.of("folder","file.txt");
        ZipHandler.writeToZip(tempZip, filePath, SAMPLE_CONTENT);

        // Create a temporary import zip file
        Path importZip = tempDir.resolve("importZip.zip");
        ZipHandler.writeToZip(importZip, Path.of("folder", "importFile.txt"), SAMPLE_CONTENT);

        // Call the importZipFile method with existing zipPath and importPath
        ZipHandler.importZipFile(tempZip, Path.of("folder"), importZip);

        // Verify that the zip file was not recreated
        assertTrue(Files.exists(tempZip));

    }

    @Test
    public void testDeleteAllSubfoldersExceptWhenZipPathDoesNotExist() throws IOException {
        // Create a temporary non-existent zip path
        Path nonExistentZipPath = tempDir.resolve("nonexistent.zip");

        // Define some dummy values for startingPath and validPaths (not relevant for this test)
        Path startingPath = Path.of("starting");
        List<Path> validPaths = new ArrayList<>();
        ZipHandler.deleteAllSubfoldersExcept(nonExistentZipPath, startingPath, validPaths);

        // Verify that the method does not perform any operations when zipPath does not exist
        assertFalse(Files.exists(nonExistentZipPath), "The method should not create the zipPath if it does not exist");
    }
}