A2-ai / dvs

Other
1 stars 0 forks source link

refactor: update names of struct fields to be more consistent #19

Closed dpastoor closed 5 months ago

dpastoor commented 5 months ago

This PR makes some breaking changes to the output formats to simplify the naming a bit.

we can simplify file_size_bytes to just size as we can document that the number corresponds to bytes + that is the only relevant. It also aligns with the common use of size in other linux utilities plus in fs for functions like dir_info().

For the timestamp, this was already an unintentional breaking change from the storage spec from the v2 go implementation, which used timestamp instead of time_stamp. The time stamp is an ambiguous name, and therefore we are going to rename it to add_time similar to other time related modifiers often present such as modification_time, creation_time, etc for normal files.

dpastoor commented 5 months ago

running a local test shows a some sensitivity related failures exist in the test suite irrespective of this. We'll need to do some cleanup and add CI before we do too much automated gating.

> devtools::test()
ℹ Testing dvs
✔ | F W  S  OK | Context
⠏ |          0 | dvs_add                                                                                                                                                Initialized empty Git repository in /tmp/RtmphkayHt/projects/add_single_file/.git/
storage directory doesn't exist
creating storage directory...
initialized storage directory: /tmp/RtmphkayHt/data/dvs/add_single_file
⠴ | 1        5 | dvs_add                                                                                                                                                [1] "Cleaning up /tmp/RtmphkayHt..."
Initialized empty Git repository in /tmp/RtmphkayHt/projects/add_multiple_file/.git/
storage directory doesn't exist
creating storage directory...
initialized storage directory: /tmp/RtmphkayHt/data/dvs/add_multiple_file
[1] "Cleaning up /tmp/RtmphkayHt..."
Initialized empty Git repository in /tmp/RtmphkayHt/projects/add_two_diff_directories/.git/
storage directory doesn't exist
creating storage directory...
initialized storage directory: /tmp/RtmphkayHt/data/dvs/add_two_diff_directories
[1] "Cleaning up /tmp/RtmphkayHt..."
Initialized empty Git repository in /tmp/RtmphkayHt/projects/no_add_meta_gitignore/.git/
storage directory doesn't exist
creating storage directory...
initialized storage directory: /tmp/RtmphkayHt/data/dvs/no_add_meta_gitignore
⠙ | 1       31 | dvs_add                                                                                                                                                [1] "Cleaning up /tmp/RtmphkayHt..."
Initialized empty Git repository in /tmp/RtmphkayHt/projects/file_dne/.git/
storage directory doesn't exist
creating storage directory...
initialized storage directory: /tmp/RtmphkayHt/data/dvs/file_dne
thread '<unnamed>' panicked at /cluster-data/user-homes/devin/.cargo/registry/src/index.crates.io-6f17d22bba15001f/extendr-api-0.6.0/src/robj/into_robj.rs:64:13:
called `Result::unwrap()` on an `Err` value: Other("at least one inputted file not found: /tmp/RtmphkayHt/projects/file_dne/data/derived/no_such_file.csv not found: No such file or directory (os error 2)")
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread '<unnamed>' panicked at src/lib.rs:70:1:
explicit panic
[1] "Cleaning up /tmp/RtmphkayHt..."
Initialized empty Git repository in /tmp/RtmphkayHt/projects/no_repo/.git/
storage directory doesn't exist
creating storage directory...
initialized storage directory: /tmp/RtmphkayHt/data/dvs/no_repo
thread '<unnamed>' panicked at /cluster-data/user-homes/devin/.cargo/registry/src/index.crates.io-6f17d22bba15001f/extendr-api-0.6.0/src/robj/into_robj.rs:64:13:
called `Result::unwrap()` on an `Err` value: Other("git repository not found: could not find git repo root; make sure you're in an active git repository: .")
thread '<unnamed>' panicked at src/lib.rs:70:1:
explicit panic
[1] "Cleaning up /tmp/RtmphkayHt..."
Initialized empty Git repository in /tmp/RtmphkayHt/projects/no_init/.git/
thread '<unnamed>' panicked at /cluster-data/user-homes/devin/.cargo/registry/src/index.crates.io-6f17d22bba15001f/extendr-api-0.6.0/src/robj/into_robj.rs:64:13:
called `Result::unwrap()` on an `Err` value: Other("configuration file not found (dvs.yaml): could not load configuration file, i.e. no dvs.yaml in directory; be sure to initiate dvs: No such file or directory (os error 2)")
thread '<unnamed>' panicked at src/lib.rs:70:1:
explicit panic
[1] "Cleaning up /tmp/RtmphkayHt..."
✖ | 1       48 | dvs_add
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Failure (test-dvs_add.R:38:3): can add a single file
is_near_time(dvs_json$time_stamp) is not TRUE

`actual`:       
`expected`: TRUE
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
⠏ |          0 | dvs_get                                                                                                                                                thread '<unnamed>' panicked at /cluster-data/user-homes/devin/.cargo/registry/src/index.crates.io-6f17d22bba15001f/extendr-api-0.6.0/src/robj/into_robj.rs:64:13:
called `Result::unwrap()` on an `Err` value: Other("metadata file not found for at least one file: missing for file")
thread '<unnamed>' panicked at src/lib.rs:526:1:
explicit panic
[1] "git_dir /tmp/RtmphkayHt/file-error/.git"
storage directory doesn't exist
creating storage directory...
initialized storage directory: /tmp/RtmphkayHt/file-error_stor_dir
[1] "proj_dir: /tmp/RtmphkayHt/file-error"
[1] "/tmp/RtmphkayHt/file-error/file14b17e430082ca.txt"
thread '<unnamed>' panicked at /cluster-data/user-homes/devin/.cargo/registry/src/index.crates.io-6f17d22bba15001f/extendr-api-0.6.0/src/robj/into_robj.rs:64:13:
called `Result::unwrap()` on an `Err` value: Other("metadata file not found for at least one file: missing for /tmp/RtmphkayHt/file-error/file14b17e430082ca.txt")
thread '<unnamed>' panicked at src/lib.rs:526:1:
explicit panic
[1] "git_dir /tmp/RtmphkayHt/file-dne/.git"
storage directory doesn't exist
creating storage directory...
initialized storage directory: /tmp/RtmphkayHt/file-dne_stor_dir
thread '<unnamed>' panicked at /cluster-data/user-homes/devin/.cargo/registry/src/index.crates.io-6f17d22bba15001f/extendr-api-0.6.0/src/robj/into_robj.rs:64:13:
called `Result::unwrap()` on an `Err` value: Other("metadata file not found for at least one file: missing for dne.txt")
thread '<unnamed>' panicked at src/lib.rs:526:1:
explicit panic
[1] "git_dir /tmp/RtmphkayHt/random/.git"
storage directory doesn't exist
creating storage directory...
initialized storage directory: /tmp/RtmphkayHt/random_stor_dir
thread '<unnamed>' panicked at /cluster-data/user-homes/devin/.cargo/registry/src/index.crates.io-6f17d22bba15001f/extendr-api-0.6.0/src/robj/into_robj.rs:64:13:
called `Result::unwrap()` on an `Err` value: Other("metadata file not found for at least one file: missing for random")
thread '<unnamed>' panicked at src/lib.rs:526:1:
explicit panic
[1] "git_dir /tmp/RtmphkayHt/file-error/.git"
storage directory doesn't exist
creating storage directory...
initialized storage directory: /tmp/RtmphkayHt/file-error_stor_dir
[1] "proj_dir: /tmp/RtmphkayHt/file-error"
[1] "/tmp/RtmphkayHt/file-error/file14b17e592fdf59.txt"
warning: no files were queued
[1] "git_dir /tmp/RtmphkayHt/status-init/.git"
thread '<unnamed>' panicked at /cluster-data/user-homes/devin/.cargo/registry/src/index.crates.io-6f17d22bba15001f/extendr-api-0.6.0/src/robj/into_robj.rs:64:13:
called `Result::unwrap()` on an `Err` value: Other("metadata file not found for at least one file: missing for file")
thread '<unnamed>' panicked at src/lib.rs:526:1:
explicit panic
[1] "git_dir /tmp/RtmphkayHt/multiple-files/.git"
storage directory doesn't exist
creating storage directory...
initialized storage directory: /tmp/RtmphkayHt/multiple-files_stor_dir
[1] "proj_dir: /tmp/RtmphkayHt/multiple-files"
[1] "git_dir /tmp/RtmphkayHt/multiple-files-glob/.git"
storage directory doesn't exist
creating storage directory...
initialized storage directory: /tmp/RtmphkayHt/multiple-files-glob_stor_dir
[1] "proj_dir: /tmp/RtmphkayHt/multiple-files-glob"
✔ |         11 | dvs_get
⠏ |          0 | dvs_init                                                                                                                                               [1] "git_dir /tmp/RtmphkayHt/first-run-init/.git"
storage directory doesn't exist
creating storage directory...
initialized storage directory: /tmp/RtmphkayHt/first-run-init_stor_dir
[1] "git_dir /tmp/RtmphkayHt/init_second_run/.git"
storage directory doesn't exist
creating storage directory...
initialized storage directory: /tmp/RtmphkayHt/init_second_run_stor_dir
thread '<unnamed>' panicked at /cluster-data/user-homes/devin/.cargo/registry/src/index.crates.io-6f17d22bba15001f/extendr-api-0.6.0/src/robj/into_robj.rs:64:13:
called `Result::unwrap()` on an `Err` value: Other("project already initialized: to change initialization settings, manually update dvs.yaml in project root")
thread '<unnamed>' panicked at src/lib.rs:48:1:
explicit panic
thread '<unnamed>' panicked at /cluster-data/user-homes/devin/.cargo/registry/src/index.crates.io-6f17d22bba15001f/extendr-api-0.6.0/src/robj/into_robj.rs:64:13:
called `Result::unwrap()` on an `Err` value: Other("git repository not found: make sure you're in an active git repository. could not find git repo root; make sure you're in an active git repository: .")
thread '<unnamed>' panicked at src/lib.rs:48:1:
explicit panic
[1] "git_dir /tmp/RtmphkayHt/no-defaults/.git"
storage directory doesn't exist
creating storage directory...
initialized storage directory: /tmp/RtmphkayHt/no-defaults_stor_dir
✔ |         11 | dvs_init
⠏ |          0 | dvs_status                                                                                                                                             [1] "git_dir /tmp/RtmphkayHt/base-case/.git"
storage directory doesn't exist
creating storage directory...
initialized storage directory: /tmp/RtmphkayHt/base-case_stor_dir
[1] "git_dir /tmp/RtmphkayHt/no-args/.git"
storage directory doesn't exist
creating storage directory...
initialized storage directory: /tmp/RtmphkayHt/no-args_stor_dir
[1] "proj_dir: /tmp/RtmphkayHt/no-args"
[1] "/tmp/RtmphkayHt/no-args/file14b17e2aded1e.txt"
[1] "/tmp/RtmphkayHt/no-args/file14b17e2aded1e.txt.dvs"
[1] "git_dir /tmp/RtmphkayHt/explicit-arg/.git"
storage directory doesn't exist
creating storage directory...
initialized storage directory: /tmp/RtmphkayHt/explicit-arg_stor_dir
[1] "proj_dir: /tmp/RtmphkayHt/explicit-arg"
[1] "/tmp/RtmphkayHt/explicit-arg/file14b17e480c70d4.txt"
[1] "/tmp/RtmphkayHt/explicit-arg/file14b17e480c70d4.txt.dvs"
[1] "git_dir /tmp/RtmphkayHt/glob-1/.git"
storage directory doesn't exist
creating storage directory...
initialized storage directory: /tmp/RtmphkayHt/glob-1_stor_dir
[1] "proj_dir: /tmp/RtmphkayHt/glob-1"
[1] "/tmp/RtmphkayHt/glob-1/file14b17eed728cb.txt"
[1] "/tmp/RtmphkayHt/glob-1/file14b17eed728cb.txt.dvs"
[1] "git_dir /tmp/RtmphkayHt/glob-2/.git"
storage directory doesn't exist
creating storage directory...
initialized storage directory: /tmp/RtmphkayHt/glob-2_stor_dir
[1] "proj_dir: /tmp/RtmphkayHt/glob-2"
[1] "/tmp/RtmphkayHt/glob-2/file14b17e63fcddb4.txt"
[1] "/tmp/RtmphkayHt/glob-2/file14b17e63fcddb4.txt.dvs"
[1] "git_dir /tmp/RtmphkayHt/status-init/.git"
thread '<unnamed>' panicked at /cluster-data/user-homes/devin/.cargo/registry/src/index.crates.io-6f17d22bba15001f/extendr-api-0.6.0/src/robj/into_robj.rs:64:13:
called `Result::unwrap()` on an `Err` value: Other("configuration file not found (dvs.yaml): could not load configuration file, i.e. no dvs.yaml in directory; be sure to initiate dvs: No such file or directory (os error 2)")
thread '<unnamed>' panicked at src/lib.rs:294:1:
explicit panic
thread '<unnamed>' panicked at /cluster-data/user-homes/devin/.cargo/registry/src/index.crates.io-6f17d22bba15001f/extendr-api-0.6.0/src/robj/into_robj.rs:64:13:
called `Result::unwrap()` on an `Err` value: Other("git repository not found: could not find git repo root; make sure you're in an active git repository: .")
thread '<unnamed>' panicked at src/lib.rs:537:1:
explicit panic
⠴ |         16 | dvs_status                                                                                                                                             [1] "git_dir /tmp/RtmphkayHt/file-error/.git"
storage directory doesn't exist
creating storage directory...
initialized storage directory: /tmp/RtmphkayHt/file-error_stor_dir
[1] "proj_dir: /tmp/RtmphkayHt/file-error"
[1] "/tmp/RtmphkayHt/file-error/file14b17e3fa3aa13.txt"
[1] "git_dir /tmp/RtmphkayHt/multiple-files/.git"
storage directory doesn't exist
creating storage directory...
initialized storage directory: /tmp/RtmphkayHt/multiple-files_stor_dir
[1] "proj_dir: /tmp/RtmphkayHt/multiple-files"
           relative_path  status size
1 file14b17e6f31a347.txt current    0
2 file14b17e33018306.txt current    0
3 file14b17e2b41939e.txt   error   NA
                                                   blake3_checksum
1 af1349b9f5f9a1a6a0404dea36dcc9499bcb25c9adc112b7cc9a93cae41f3262
2 af1349b9f5f9a1a6a0404dea36dcc9499bcb25c9adc112b7cc9a93cae41f3262
3                                                             <NA>
                  add_time saved_by message
1 2024-06-09T01:19:13.123Z    devin        
2 2024-06-09T01:19:13.123Z    devin        
3                     <NA>     <NA>    <NA>
                                          absolute_path          error
1 /tmp/RtmphkayHt/multiple-files/file14b17e6f31a347.txt           <NA>
2 /tmp/RtmphkayHt/multiple-files/file14b17e33018306.txt           <NA>
3 /tmp/RtmphkayHt/multiple-files/file14b17e2b41939e.txt file not added
                                                error_message
1                                                        <NA>
2                                                        <NA>
3 metadata file not found - add the file to dvs to get status
                                                  input
1                                                  <NA>
2                                                  <NA>
3 /tmp/RtmphkayHt/multiple-files/file14b17e2b41939e.txt
[1] "git_dir /tmp/RtmphkayHt/multiple-files-glob/.git"
storage directory doesn't exist
creating storage directory...
initialized storage directory: /tmp/RtmphkayHt/multiple-files-glob_stor_dir
✔ |         25 | dvs_status
⠏ |          0 | integrated1                                                                                                                                            [1] "git_dir /tmp/RtmphkayHt/update/.git"
storage directory doesn't exist
creating storage directory...
initialized storage directory: /tmp/RtmphkayHt/update_stor_dir
[1] "inputted time: "
[1] "current time: "
[1] "time_difference: "
[1] "inputted time: "
[1] "current time: "
[1] "time_difference: "
difftime of length 0
[1] "inputted time: 2024-06-09 01:19:13.230247"
[1] "current time: 2024-06-09 01:19:13.230247"
[1] "time_difference: 0.000227689743041992"
[1] "inputted time: 2024-06-09 01:19:13.235493"
[1] "current time: 2024-06-09 01:19:13.235493"
[1] "time_difference: 5.0004575252533"
✖ | 3        8 | integrated1 [5.1s]                                                                                                                                     
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Failure (test-integrated1.R:18:5): add fn updates a file
near_system_time(init_add_time) is not TRUE

`actual`:       
`expected`: TRUE
Backtrace:
    ▆
 1. ├─withr::with_dir(...) at test-integrated1.R:4:3
 2. │ └─base::force(code)
 3. └─testthat::expect_true(near_system_time(init_add_time)) at test-integrated1.R:18:5

Failure (test-integrated1.R:43:5): add fn updates a file
near_system_time(update_time) is not TRUE

`actual`:       
`expected`: TRUE
Backtrace:
    ▆
 1. ├─withr::with_dir(...) at test-integrated1.R:4:3
 2. │ └─base::force(code)
 3. └─testthat::expect_true(near_system_time(update_time)) at test-integrated1.R:43:5

Failure (test-integrated1.R:47:5): add fn updates a file
init_add_time < update_time is not TRUE

`actual`:       
`expected`: TRUE
Backtrace:
    ▆
 1. ├─withr::with_dir(...) at test-integrated1.R:4:3
 2. │ └─base::force(code)
 3. └─testthat::expect_true(init_add_time < update_time) at test-integrated1.R:47:5
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
⠏ |          0 | integrated2                                                                                                                                            [1] "git_dir /tmp/RtmphkayHt/update/.git"
storage directory doesn't exist
creating storage directory...
initialized storage directory: /tmp/RtmphkayHt/update_stor_dir
✔ |          7 | integrated2
⠏ |          0 | integrated3                                                                                                                                            [1] "git_dir /tmp/RtmphkayHt/no-op/.git"
storage directory doesn't exist
creating storage directory...
initialized storage directory: /tmp/RtmphkayHt/no-op_stor_dir
✔ |          5 | integrated3
⠏ |          0 | integrated4                                                                                                                                            [1] "git_dir /tmp/RtmphkayHt/update/.git"
storage directory doesn't exist
creating storage directory...
initialized storage directory: /tmp/RtmphkayHt/update_stor_dir
✔ |          7 | integrated4

══ Results ═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════
Duration: 6.5 s

── Failed tests ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Failure (test-dvs_add.R:38:3): can add a single file
is_near_time(dvs_json$time_stamp) is not TRUE

`actual`:       
`expected`: TRUE

Failure (test-integrated1.R:18:5): add fn updates a file
near_system_time(init_add_time) is not TRUE

`actual`:       
`expected`: TRUE
Backtrace:
    ▆
 1. ├─withr::with_dir(...) at test-integrated1.R:4:3
 2. │ └─base::force(code)
 3. └─testthat::expect_true(near_system_time(init_add_time)) at test-integrated1.R:18:5

Failure (test-integrated1.R:43:5): add fn updates a file
near_system_time(update_time) is not TRUE

`actual`:       
`expected`: TRUE
Backtrace:
    ▆
 1. ├─withr::with_dir(...) at test-integrated1.R:4:3
 2. │ └─base::force(code)
 3. └─testthat::expect_true(near_system_time(update_time)) at test-integrated1.R:43:5

Failure (test-integrated1.R:47:5): add fn updates a file
init_add_time < update_time is not TRUE

`actual`:       
`expected`: TRUE
Backtrace:
    ▆
 1. ├─withr::with_dir(...) at test-integrated1.R:4:3
 2. │ └─base::force(code)
 3. └─testthat::expect_true(init_add_time < update_time) at test-integrated1.R:47:5

[ FAIL 4 | WARN 0 | SKIP 0 | PASS 122 ]