Open Sicheng2000 opened 4 months ago
Very detailed response. Thank you!
On #2: Consider the tables in Chapter 10 to help you decide on which statistic to calculate for the statistical design of your hypothesis. Once you know that, the rest of the workflow is very similar for (almost) all the other tests.
On #3: Yes, the ENNTT is very large. It will take a lot of computer processing resources. And yes, if you accidentally (try to) push the data to GitHub you will get an error --GitHub is not made to store large files. There is a way around it, but in this case it is just best to add the data/datasets to the .gitignore
file.
On #4: Down sampling is not without it's potential issues, but the down sampling reduces the dataset size while attempting to maintain a similar variation pattern as the full dataset.
@francojc
cache: true
. In addition, I forgot to adddata/original/*
in.gitignore
the first time it made me hard to push to GitHub (https://stackoverflow.com/questions/51764180/file-too-large-added-to-git-ignore-but-still-trying-to-add) even though I tried to downloadgit lf
(https://stackoverflow.com/questions/48734119/git-lfs-is-not-a-git-command-unclear). For the model choice, I based it on the same model used in the textbook (https://qtalr.github.io/qtalrkit/articles/recipe-10.html). Because the translation file is too big toread_html
, so I could not visualize the data. Although there is a function we could use, it still does not work because it takes too much time to run, so my Rstudio just terminates automatically.step_downsample()
. However, it seemsstep_downsample()
will remove rows of data which may influence accuracy. So I searched for information about when should usestep_downsample()
(https://medium.com/@rithpansanga/choosing-the-right-size-a-look-at-the-differences-between-upsampling-and-downsampling-methods-daae83915c19). It seems for large dataset, downsampling is preferred. The link I provided talks about downsampling and upsampling's pros and cons and the when to use them.