Wanted to open a discussion about the intended behaviour of SPLIT_TIME_THRESHOLD in conjunction with SPLIT_OUTPUT_FILE.
Context
In my CI setup, the original timings.json file is kept in cypress/e2e/<sub-project>/timings.json, and I am setting the SPLIT_TIME_THRESHOLD to be 0.2.
I am also using SPLIT_OUTPUT_FILE to save the updated timings for each parallel node into their own artifacts directory like so:
artifacts/<sub-project>/split-timings/node-<1,2,3,etc>/timings.json. These are then merged back together by a later merge job using the cypress-split-merge binary (which is working great btw!)
Expected Behaviour (in my head)
The behaviour I was expecting here was that the SPLIT_OUTPUT_FILE would not include updated timings unless the threshold is met. If the threshold was not met, it would either:
not write out an updated timings.json at all (this might not work well when merging later)
or it would write to the node-X output file, but it would be the original times of the test cases and not the updated ones. in this case, when merging the updated timings, Git can realise there is no change to the merged timings file and not bother committing.
Actual Behaviour
Despite the changes not meeting the threshold, it writes the updated timings to the SPLIT_OUTPUT_FILE anyway:
Wanted to open a discussion about the intended behaviour of
SPLIT_TIME_THRESHOLDin conjunction withSPLIT_OUTPUT_FILE.Context In my CI setup, the original
timings.jsonfile is kept incypress/e2e/<sub-project>/timings.json, and I am setting theSPLIT_TIME_THRESHOLDto be0.2.I am also using
SPLIT_OUTPUT_FILEto save the updated timings for each parallel node into their own artifacts directory like so:artifacts/<sub-project>/split-timings/node-<1,2,3,etc>/timings.json. These are then merged back together by a later merge job using thecypress-split-mergebinary (which is working great btw!)Expected Behaviour (in my head) The behaviour I was expecting here was that the
SPLIT_OUTPUT_FILEwould not include updated timings unless the threshold is met. If the threshold was not met, it would either:node-Xoutput file, but it would be the original times of the test cases and not the updated ones. in this case, when merging the updated timings, Git can realise there is no change to the merged timings file and not bother committing.Actual Behaviour Despite the changes not meeting the threshold, it writes the updated timings to the
SPLIT_OUTPUT_FILEanyway:I can see in the code it's because
OUTPUT!== theINPUTfile, so we write it anyway: https://github.com/bahmutov/cypress-split/blob/16f7a78c5183c5ccf15cdecab29b0888d368348b/src/index.js#L256As a result of the above, later in our merge job the spec timings are being updated no matter what the threshold is.
Question for you What would be the ideal way for
SPLIT_OUTPUT_FILEto respectSPLIT_TIME_THRESHOLD? Would love to help hop on a solution!