search

ian-james / IFS

Immediate Feedback System for student programming assignments and essay writing
MIT License
5 stars 6 forks source link

IFS -> Submissions of Large files occassional drops a file. #143

Closed ian-james closed 6 years ago

ian-james commented 6 years ago

Identified by John

Occasionally, IFS will drop a file if a number of files are submitted and 1+ are large. Tended to be the large file that was dropped.

Solution: The issue appeared to stem from the asynchronous creation of the users/submission folder which was being erased and recreated between file uploads. The folder creation needs to happen once at submission and then upload files.

ryoung2512 commented 6 years ago

John and I looked into it. We were able to fix the problem of having the files deleted, by using ensureDir to ensure the directory exists if not make it. As well as timestamps for the folder directory. With that being said another bug has emerged which is that it will create a second folder a millisecond later sometimes. This needs to be addressed but is difficult cuz we have to sift through a bunch of files.

We will probably wait until you are back to get your input on how some of it works.

ian-james commented 6 years ago

Something like the code you're talking about previously exists in GitHub. Basically, you want to round or chop the timestamp based on a reasonable assumption. As students, aren't submitting every second but maybe every 5 seconds. Thus, really a timestamp has to a minimum chop the ms and possibly even the 1st digit of seconds. There are other ways to do this of course just saying that exists if you look back in history.

Realistically, I'm going to push for saving all submission again in an ethics document soon. So this only requires minor adjustments to the saving structure of submissions.

In addition, we need to perhaps move the clearing of the directory or creating of a new directory if saving submissions outside of the upload process. So that it only occurs once and then we don't have these millisecond type issu

ryoung2512 commented 6 years ago

Sorry, forgot to update this. We were able to fix it by passing timestamp through the header client side. We currently however always store the files. This can be changed.

ian-james commented 6 years ago

Yeah, just heard. We're currently breaking the ethics rules but that isn't important until we deploy.

So the current dev branch is up-to-date with the anticipated ethics document.

I've indicated to John as well, but we will IFS to have researcher settings to toggle these features based on who is running what experiment. So this feature will need to have another option of single-safe that is determined by an options file set when IFS first runs.

On Wed, Jul 18, 2018 at 5:14 PM, Rhys Young notifications@github.com wrote:

Sorry, forgot to update this. We were able to fix it by passing timestamp through the header client side. We currently however always store the files. This can be changed.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ian-james/IFS/issues/143#issuecomment-406076609, or mute the thread https://github.com/notifications/unsubscribe-auth/AEOnbhv4GvAsI1D00SjIoxeANMLKMc6cks5uH6VAgaJpZM4VCx0Q .