Videos disappearing while doing other stuff

[susanodd]

          @Jetske I'm having a different error.

The id videoplayer is duplicate in Gloss Detail. The sentence video has the same id as the gloss video and it can't run the javascript to load the video.

    function reloadVideo() {
        var video_player = document.getElementById("videoplayer");
        video_player_src = video_player.src;
        video_player_src = "";
        video_player.src = video_player_src;
        video_player.type = "video/mp4";
        video_player.load();
    }

I'm not able to upload a new video for a gloss because it has a bug with this and the javascript isn't completing.

Originally posted by @susanodd in https://github.com/Signbank/Global-signbank/issues/1326#issuecomment-2370764589

The code stops with an error in the javascript console attempting to run the above. (#1065)

[susanodd]

I needed to remove the code that "reloads" an updated video. It doesn't work on Safari. And for some reason, the uploaded videos are disappearing. Also the files are disappearing from iCloud. (#1065)

This could be a transaction problem, or some other code is deleting files? Or there are more shared IDs in the Gloss template that the javascript uses.

I can't figure out what is wrong. I just noticed that after a couple of reloading the page (while writing code I need to keep doing this), the new video files for perspective had disappeared. But they still show in the Admin. But the files had disappeared.

[susanodd]

This is also going to cause problems with displaying the perspective videos. They need to swap out the gloss video. But then where should they "exist" on the page when swapped out? They can't keep being reloaded. Or can they? The videos on the page appear as "src" links. But are they still loaded if they're inside of invisible elements?

I don't think the browser is intelligent to layer them behind each other say with z-index. And they can't really "hide" anywhere. But do we want video files to keep reloading from the hard drive and swapping out? This probably won't work on some browsers. Safari is having a problem. And files are disappearing. Safari hides advertising and blocks these from appearing. We don't do anything about Cookie Permissions on Signbank.

[susanodd]

Question (@Woseseltops @vanlummelhuizen @Jetske )

This code in model GlossVideo (and others):

    def get_absolute_url(self):

        return self.videofile.url

There is no url field in this object. (I put in print statements to see the fields it has elsewhere)

Can this account for the erratic behaviour? Other methods do this, too, as well as "path" which also does not exist.

This happens on PyCharm runserver (both MacOS and Ubuntu) PyCharm seems to be strict in running code, since it shows warnings about the above, indicating that the field does not exist, in the code editor. Could it be that Apache Django just coerces things to string if it can't find the field?

[Jetske]

@susanodd my first thought was also to use z-index. Otherwise, maybe make the perspective video hidden/visible in css depending on what is selected.

And about .url it looks like there are django APIs for filefields: https://docs.djangoproject.com/en/dev/topics/files/

[vanlummelhuizen]

There is no url field in this object. (I put in print statements to see the fields it has elsewhere)

The docs say it should have a url variable: https://docs.djangoproject.com/en/5.1/ref/models/fields/#filefield-and-fieldfile.

Quick test in a Django shell:

>>> from signbank.video.models import *
>>> GlossVideo.objects.first()
<GlossVideo: glossvideo/NGT/KA/KALENDER-A-753.mp4>
>>> GlossVideo.objects.first().videofile
<FieldFile: glossvideo/NGT/KA/KALENDER-A-753.mp4>
>>> GlossVideo.objects.first().videofile.url
'/media/glossvideo/NGT/KA/KALENDER-A-753.mp4'
>>> GlossVideo.objects.first().videofile.path
'/home/micha/docs/ru/signbank/global-signbank/writable/glossvideo/NGT/KA/KALENDER-A-753.mp4'

[susanodd]

We're not using Django 5.1

It does not work like this for PyCharm.

[vanlummelhuizen]

https://docs.djangoproject.com/en/4.2/ref/models/fields/#filefield-and-fieldfile

It does not work like this for PyCharm.

What does not work?

[susanodd]

https://docs.djangoproject.com/en/4.2/ref/models/fields/#filefield-and-fieldfile

It does not work like this for PyCharm.

What does not work?

That it is deleting files mysteriously. I will put in more print statements to try to detect. Or more try-except and actually use the exception string. It could be that other stuff is going wrong but the code keeps running after doing a pass or setting some default. I came across lots of places where in very old code the default dataset is being used if it's missing. That would also cause a wrong path-folder. You're right, I need to find proof of what it's actually doing. So something happens before it tries to create the image.

[susanodd]

[NOTE TO SELF: EXAMINE CODE TO FIND QUIRKS] I thought of something else that could be happening. Django stores the files, but there are also files in the storage system. When a new file is uploaded, Django does not know that this is the same file. (A new object is created, the old file may or may not be moved to a different name. The reversion is there.) But to replace an "object" so the gloss video file in the object corresponds to the file on disk is different. (I did this in some code, but had to override the normal video stuff.) Note to self: document this.

[susanodd]

@vanlummelhuizen

Just got this error (screenshot below):

[susanodd]

Okay, this is a total mystery. This happens with PyCharm runserver running Signbank.

I upload with success the videos.

I do other stuff on the computer, edit code, search things in other browser windows, but do nothing with signbank.

When I return to my browser page, I see that the videos are not visible anymore. I reload the page and it's showing the "missing video blank video" instead of the video. The files are also gone from writable.

Does Django do stuff with videos when you're not doing anything? Is there a queue somewhere that hasn't finished? Does reversion do stuff in the background? Has it somehow signaled commands in the past that are not executed yet and it's "doing stuff" that happened days ago? I uploaded the videos yesterday. Then they were gone today. Then I uploaded them again and it was fine. Then 30 minutes or so later they were gone again.

@vanlummelhuizen any ideas? I am not "running anything, not doing any commands in signbank". It just happens.

I can't debug this because I'm not "running any code" when it happens.

[susanodd]

Here is the writable:

susaneven@MacBook-Air-van-Susan writable % ls -l glossvideo/tstMH/PE
total 12416
-rw-r--r--  1 susaneven  staff  2021569 24 sep 14:47 PERSPECTIEF-36430.mp4
-rw-------  1 susaneven  staff  2213202 27 sep 09:57 PERSPECTIEF-36430_left.mp4
-rw-------  1 susaneven  staff  2113652 27 sep 09:58 PERSPECTIEF-36430_right.mp4
susaneven@MacBook-Air-van-Susan writable % ls -l glossvideo/tstMH/PE
total 3952
-rw-r--r--  1 susaneven  staff  2021569 24 sep 14:47 PERSPECTIEF-36430.mp4
susaneven@MacBook-Air-van-Susan writable % 

I did nothing at all with Signbank nor with PyCharm. The files just vanished. It's 10:38 now.

[susanodd]

Related, but about GitHub. This also happens on Ubuntu when I drag a file into the comments. The file is visible for a while, then it turns into a bold heading and it's not a screenshot anymore. It says it's a private source.

Did the EU get browsers to not allow uploading things or something? It seems this is recent change to browsers. For GitHub I need to copy paste. Not drag and drop. Now it seems that Safari is also doing this.

[susanodd]

I changed to code to use "shutil.copy" instead of "shutil.move" but that does not solve the problem. The files still disappear after some minutes. Is the operating system prohibiting that the site can "steal" uploaded files? Like private data or something? That GitHub does the same thing seems suspicious. Is this why @Woseseltops suggested to always zip the files? The citation image also disappears.

[vanlummelhuizen]

To be clear, this only happens on your local computer using Pycharm and there is no problem on the server, correct? If so, it is very hard for us help you. One thing you could try is to set up a new local instance and see whether the problem goes away.

I am not "running anything, not doing any commands in signbank". It just happens.

There is something doing this. It's just that you don't know what.

Did the EU get browsers to not allow uploading things or something? It seems this is recent change to browsers.

No. This seems like a wild speculation. We should stick to the facts here.

Does Django do stuff with videos when you're not doing anything? Is there a queue somewhere that hasn't finished? Does reversion do stuff in the background? Has it somehow signaled commands in the past that are not executed yet and it's "doing stuff" that happened days ago? I uploaded the videos yesterday. Then they were gone today. Then I uploaded them again and it was fine. Then 30 minutes or so later they were gone again.

No.

[susanodd]

I just changed it to "shutil.copyfile" but that also disappears.

I am not sure yet if it is okay on the Apache server. It's becoming difficult to develop code on PyCharm.

I have the perspective videos working now. But then they keep disappearing.

[susanodd]

There is something doing this. It's just that you don't know what.

Correct. (It's causing stuff to disappear while I am developing code and testing it.)

I guess I can assume the code is correct because it worked once. (But really?) (The tests also just do operations once. They don't test whether things are persistent.)

[susanodd]

The worrying thing or what I'm hoping is not happening, is that a transaction isn't being rolled back that I don't know about.

[susanodd]

The videos also disappear when os.rename is used.

[susanodd]

@vanlummelhuizen can this be related to the original being a video that's in an unzipped archive? It seems to keep being a symbolic link wherever I put it. I got the perspective videos in a zip file and unzipped it. I can't find anything helpful about how to make it "not" be a symbolic link. This would be the OS then.

[susanodd]

The citation image also disappears. That wasn't a symbolic link because it was created when a new video was uploaded. The videos and image don't disappear right away.

How can I debug code if I can't tell if there are bugs causing this?

There is some video code that runs as signals triggered by "video.save()". This code seems to run later. (On PyCharm maybe this code is done much later and is causing things to disappear.)

I suspect now that there are too many "signal" calls caused my extra "save" method calls to a Gloss Update in order to update the timestamp. This was done because of the package API needs the timestamps to be updated a lot in order to tell when changes have been made. So lots of Gloss update code does "save" now. The saves trigger signals that also apply to the video code. It seems it's calling "move_videos" lots of times.

[susanodd]

@vanlummelhuizen I think this is being called an excessive amount of times:

https://github.com/Signbank/Global-signbank/blob/6242cc47de593c3debcd78da7a2f2a25f97f0f62/signbank/video/models.py#L1011-L1013

(Because the API package functionality as used by UvA wants to know all recent changes to glosses, lots of save timestamp code has been added because if related classes are updated, this was not indicated that the gloss was updated. It looks like this code is now called a lot now, I put print statements in it. This should only be called if a lemma or annotation has been changed, or a dataset has been renamed, not every time a gloss has been updated.)

[susanodd]

Okay, I added a field to the signal methods so that videos are only moved if an argument is specified.

But this also does not solve the disappearing files.

It does keep from excessive calls of the methods.

[susanodd]

The most recent version uses:

os.path.realpath

applied to the temporary uploaded file, videofile.file.name (there is no url or path field on this object).

So far, the videos have held for several hours and have not disappeared.

[susanodd]

And they're gone.

susaneven@MacBook-Air-van-Susan tstMH % ls -l we
total 12896
-rw-r--r--@ 1 susaneven  staff        0 31 jan  2024 wesseltest3-44207_small.mp4
-rw-r--r--  1 susaneven  staff  2021569  1 okt 08:00 wesseltest4-37052.mp4
-rw-r--r--@ 1 susaneven  staff  2213202  1 okt 08:04 wesseltest4-37052_left.mp4
-rw-r--r--@ 1 susaneven  staff  2113652  1 okt 08:05 wesseltest4-37052_right.mp4
-rw-r--r--  1 susaneven  staff   122035 11 jun 10:27 wesseltest6-3117_nme_1.mp4
-rw-r--r--  1 susaneven  staff   119608 11 jun 17:35 wesseltest6-3117_nme_1_G3VgjDQ.mp4
susaneven@MacBook-Air-van-Susan tstMH % ls -l we
total 4432
-rw-r--r--@ 1 susaneven  staff        0 31 jan  2024 wesseltest3-44207_small.mp4
-rw-r--r--  1 susaneven  staff  2021569  1 okt 08:00 wesseltest4-37052.mp4
-rw-r--r--  1 susaneven  staff   122035 11 jun 10:27 wesseltest6-3117_nme_1.mp4
-rw-r--r--  1 susaneven  staff   119608 11 jun 17:35 wesseltest6-3117_nme_1_G3VgjDQ.mp4
susaneven@MacBook-Air-van-Susan tstMH % 

They disappeared not doing anything at all. Not using signbank and not using PyCharm. Just doing "ls -l" in the console. The second ls-command was done five minute after the first.

[susanodd]

The videos survived! There was a setting for videos that was causing the files not to be deleted even if the model needed to delete them.

susaneven@MacBook-Air-van-Susan we % ls -l
total 15600
-rw-r--r--  1 susaneven  staff  2021569  1 okt 08:00 wesseltest4-37052.mp4
-rw-r--r--  1 susaneven  staff  2213202  1 okt 17:25 wesseltest4-37052_left.mp4
-rw-r--r--  1 susaneven  staff  1504698  1 okt 15:35 wesseltest4-37052_nme_3.mp4
-rw-r--r--  1 susaneven  staff  2113652  1 okt 17:24 wesseltest4-37052_right.mp4
-rw-r--r--  1 susaneven  staff   122035 11 jun 10:27 wesseltest6-3117_nme_1.mp4
susaneven@MacBook-Air-van-Susan we % ls -l
total 15600
-rw-r--r--  1 susaneven  staff  2021569  1 okt 08:00 wesseltest4-37052.mp4
-rw-r--r--  1 susaneven  staff  2213202  1 okt 17:25 wesseltest4-37052_left.mp4
-rw-r--r--  1 susaneven  staff  1504698  1 okt 15:35 wesseltest4-37052_nme_3.mp4
-rw-r--r--  1 susaneven  staff  2113652  1 okt 17:24 wesseltest4-37052_right.mp4
-rw-r--r--  1 susaneven  staff   122035 11 jun 10:27 wesseltest6-3117_nme_1.mp4
susaneven@MacBook-Air-van-Susan we %