search

cta-wave / dpctf-deploy

Repo for Docker deployment. This is basically the repository you need to deploy the Test Runner (and later the Test Execution Framework) locally. It will make the deployment and configuration of all components much easier
Other
2 stars 5 forks source link

./import.sh fails using WSL/Powershell (Terminal) #84

Open gitwjr opened 4 months ago

gitwjr commented 4 months ago

@louaybassbouss @FritzHeiden After a complete rebuild (reinstall WSL w/Ubuntu 20.04 dstro), Powershell 7, Docker Desktop, and following the current QS guide ./build.sh master latest creates the docker-deploy folder but when I run ./import.sh the Terminal flashes a /usr/bin/bash --login and fails saying the device is not a TTY.

image

I can run .\import.bat but that leads to other issues. I've tried docker set to "enable integration with my default WSL distro" with and without "enable integration with additional distros (Unbuntu20.04"). No difference. The only image in docker is dpctf (tag "latest"). No containers show up. The QS Guide says use ./import.sh for WSL.

mlasak commented 4 months ago

Pls have a look what i've done on a win10 with wsl2 to successfully execute test runner without docker for desktop (installed docker inside wsl2/ubuntu22.04.3 LTS)

https://youtu.be/zYi-uaRAYFg

PS C:\Users\local_root> wsl -v
WSL-Version: 2.0.9.0
Kernelversion: 5.15.133.1-1
WSLg-Version: 1.0.59
MSRDC-Version: 1.2.4677
Direct3D-Version: 1.611.1-81528511
DXCore-Version: 10.0.25131.1002-220531-1700.rs-onecore-base2-hyp
Windows-Version: 10.0.19045.4046
jpiesing commented 4 months ago

Hang on a moment.

Is the combination where part of the system is running in the commercial Docker Desktop and part in WSL outside Docker Desktop in any way supported?

If that's what Bill was trying to do here (perhaps without realising) then IMHO the answer should be that this mixed deployment is really, really not recommended.

There are legitimate reasons for using WSL2 without Docker Desktop but IMHO it should be all or nothing and not mixed.

Is build.sh expected to work with Docker Desktop? From my limited understanding, if you're using Docker Desktop then you should be using build.bat and import.bat. Is that the reason why import.bat isn't working?

If I'm correct then the documentation should be changed something like this ...

From

Linux / macOS / WSL2: To Linux / macOS / Windows using WSL2 without Docker Desktop

and from

Windows: To Windows using Docker Desktop

FritzHeiden commented 4 months ago

From what I experienced, running docker in WSL and outside with Docker-Desktop is not a problem.

If I understand this correctly, Bill installed WSL with Ubuntu and tries to run the build.sh in a powershell. This, however, cannot work, as powershell is not bash. In the documentation the commands for Linux, macOS and WSL are the same because they are all executed using a shell, like bash. In order to do that with WSL you have to actually run wsl in powershell. This will put you in a shell that runs on WSL. From here you can run the shell scripts:

image

jpiesing commented 4 months ago

From what I experienced, running docker in WSL and outside with Docker-Desktop is not a problem.

https://github.com/cta-wave/dpctf-deploy/issues/63 might suggest otherwise.

gitwjr commented 4 months ago

How do I run WSL in powershell? I uninstalled and reinstalled WSL with ubuntu 20.04 per the microsoft instructions and restarted the computer. I start the LSXX service. I then followed the QS guide cloning dpctf-deploy using git-cmd and running ./build.sh in a powershell (Terminal) as Admin all per the QS. I see the ./build.sh open a new window which appears to be a bash, runs to the end and then exits back to Terminal. The build creates the dpctf latest image in docker and the dpctf-deploy folder. THEN I run ./import.sh which opens a bin-bash which flashes the bin-bash so fast I had to record it and step frame by frame to see what was running). It runs for just a moment (initially says "DATA SIZE warning: This script will download a lot of data!") then shows the "TTY" error and fails back to Terminal window with no information. I am currently using Powershell 7 but had been using PS 5. I installed the latest version to see if that was the issue. It appears something in the script differs from the ./build.sh script causing the TTY error.

gitwjr commented 4 months ago

When I run "wsl --list --verbose" I get PS C:\Users\brose> wsl --list --verbose NAME STATE VERSION

Do I need to change the default to docker-desktop or docker-desktop-data?

gitwjr commented 4 months ago

Just tried changing default to docker-desktop and running in Terminal and in Powershell. No change. Also tried docker-desktop-data as default. No change. PS C:\Users\brose> cd dpctf-deploy PS C:\Users\brose\dpctf-deploy> ./import.sh PS C:\Users\brose\dpctf-deploy> wsl --list --verbose NAME STATE VERSION

gitwjr commented 4 months ago

@FritzHeiden ./build.sh master latest runs fine and creates the dpctf image in Docker. it is ./import.sh that does not run.

FritzHeiden commented 4 months ago

@gitwjr If you run commands and script directly in the powershell, then use the ones labeled as "Windows:" in the QS. This requires Docker-Desktop.

If you want to use WSL, open powershell, type wsl and hit enter. Now the prompt in front of the cursor changed. (See the screenshot here). Now you can run the commands and scripts labeled as "Linux / macOS /WSL2".

FritzHeiden commented 4 months ago

If you are inside WSL (again type wsl in powershell and hit enter) you can't access Docker-Desktop. So if you did not install docker inside WSL, then it is not available and thus any docker related scripts and commands will fail.

mlasak commented 4 months ago

i found this quite useful: https://dev.to/bowmanjd/install-docker-on-windows-wsl-without-docker-desktop-34m9 prior to this i simply run on a fresh win10 in powershell

wsl --install

what installed ubuntu 22 (see above)

gitwjr commented 4 months ago

I installed docker using the instructions at https://docs.docker.com/desktop/install/windows-install/ WSL2 was uninstalled then reinstalled prior to reinstalling docker per the above instructions. As far as I know docker should be installed in WSL. Not to sound too ignorant but how do I know docker is inside of WSL? The ./build.sh ran fine in a windows Terminal. I've tried to run import.sh in Terminal, and in Powershell. If I run Powershell directly it comes up in PS C:\WINDOWS\system32. I then CD to dpctf-deploy and run import.sh - FAILS. If I run "wsl" in the powershell it comes up Bill-Lenovo-Laptop:/mnt/host/c/WINDOWS/system32#. I assume I need to cd to dpctf-deploy but it seems I am stuck in the /mnt/hosts/c/... directory and can't get to dpctf-deploy outside of "host" and so it can't find the ./import.sh file.

PS C:\Users\brose> wsl Bill-Lenovo-Laptop:/mnt/host/c/Users/brose# cd dpctf-deploy Bill-Lenovo-Laptop:/mnt/host/c/Users/brose/dpctf-deploy# ./import.sh -ash: ./import.sh: not found Bill-Lenovo-Laptop:/mnt/host/c/Users/brose/dpctf-deploy#

I also changed the default for wsl using --set-default to docker-desktop. no change

jpiesing commented 4 months ago

@gitwjr My understanding is that, if you're using Docker Desktop, then, to be consistent, you should be using Powershell, build.bat and import.bat. build.sh and import.sh are for people using Linux and docker that is not Docker Desktop.

Happy to be corrected.

gitwjr commented 4 months ago

So I should rebuild by deleting dpctf-deploy folder and the Docker dpctf image and rebuild using .\build.bat master latest and .\import.bat? I probably should clear cache on docker as well?

mlasak commented 4 months ago

Agree with @jpiesing, either

Disagree with your approach @gitwjr, if you delete dpctf-deploy folder you can not expect any of the files needed to be present! No need to delete dpctf-deploy folder ever

If you decide for WSL2 (which i recommend for the test runner on windows), please execute the following steps and state which is not successful for you.

(Ideally on a fresh Win10 Build 19041 or higher) Open powershell and then type

wsl --install
wsl -v
# should show you versions, more info: https://learn.microsoft.com/de-de/windows/wsl/install
wsl 
# now you are in ubuntu! and install docker as described in https://dev.to/bowmanjd/install-docker-on-windows-wsl-without-docker-desktop-34m9
. /etc/os-release
curl -fsSL https://download.docker.com/linux/${ID}/gpg | sudo tee /etc/apt/trusted.gpg.d/docker.asc
echo "deb [arch=amd64] https://download.docker.com/linux/${ID} ${VERSION_CODENAME} stable" | sudo tee /etc/apt/sources.list.d/docker.list
sudo apt update
sudo apt install docker-ce docker-ce-cli containerd.io
sudo usermod -aG docker $USER
exit
# now you are again in powershell, enter wsl/ubuntu again and check docker is present as in my video https://youtu.be/zYi-uaRAYFg
wsl
docker
docker-desktop
git

If all above steps were ok, then proceed with instructions in dpctf-deploy REAMDE.md and make sure you are always inside wsl/ubuntu.

good luck and i hope this helps further.

gitwjr commented 4 months ago

@jpiesing @louaybassbouss @yanj-github @FritzHeiden Conclusion:: QS Guide should say use ....bat not ./...sh to run the build commands when using WSL2/Windows. It works! ./...sh does not for import.

I am running WSL2/Windows 11-Pro and built according to the QS guide except I followed Jon's suggestion and used ....bat not ./...sh for the commands in Powershell (clone in git cmd). I used .\build.bat master latest and .\import.bat (and to build dof) in Powershell. No problems encountered with the rebuild at all.

I then did 4 tests/recordings using .\analyse-recordings.bat. All went to completion and posted results. FYI, at the time I had docker-desktop set as default per "wsl --list --verbose". Not sure if that made a difference but it didn't help when I was previously trying to use .\import.sh. The recordings all had failures. Each of them reported 1 corrupted frame (between 88 and 201 frames from the end) according to the powershell end message. I am not too concerned about the failures. Appears to be my pixel camera, recording setup, or an interaction between my camera and monitors. (I did try using different monitor for one recording). 3 were recorded at 60fps/1080P. Number 1) below was recorded at at 60fps/4k uhd. The results of the failures are as follows (all other tests were PASS).

1) /cfhd_12.5_25_50-local/fullscreen-playback-of-switching-sets__ss1-2.html: "status": "FAIL", "message": " Mid frame number tolerance is 10. Following frames are missing in playout 7: 221 229 241 295 334 367 374 394 Following frames are missing in playout 6: 408 428 435 454 528 534 547 554 561 568 Playout 3 starting frame found is 2302, expected to start from 2301. Total of missing frames is 18.", "name": "[OF] Every video frame S[k,s] shall be rendered and the video frames shall be rendered in increasing presentation time order."

2) /cfhd_12.5_25_50-local/sequential-track-playback__t1.html (10 missing frames - Pass) /cfhd_12.5_25_50-local/sequential-track-playback__t1.html "status": "FAIL", "message": "Playback duration 29934.7ms does not match expected duration 30000.0ms +/- tolerance of 50ms. Detected duration is different by 65.3ms. Allowed tolerance is 50ms and duration frame tolerance is 0. Starting missing frame number is 0. Ending missing frame number is 0.",

3) /cfhd_12.5_25_50-local/fullscreen-playback-of-switching-sets__ss1-2.html "status": "FAIL", "message": " Mid frame number tolerance is 10. Following frames are missing in playout 7: 210 230 242 262 395 Following frames are missing in playout 6: 409 449 455 489 547 569 595 Playout 3 starting frame found is 2302, expected to start from 2301. Total of missing frames is 12.",

4) /cfhd_12.5_25_50-local/sequential-track-playback__t1.html status": "FAIL", "message": "Playback duration 29927.51ms does not match expected duration 30000.0ms +/- tolerance of 50ms. Detected duration is different by 72.49ms. Allowed tolerance is 50ms and duration frame tolerance is 0. Starting missing frame number is 0. Ending missing frame number is 0.",

gitwjr commented 4 months ago

@mlasak Sorry if I wasn't clear. I delete the dpctf-deploy folder and the docker images and container prior to each new rebuild so there are no remnants of files and I know the complete build is clean and up to date. Of course I have to change config.json, docker-compose.yml, and add the observation-config.ini files each time but it eliminates one possible source of problems. In this case the dpctf-deploy was built using ./build.sh and I wanted to test a complete build using the .bat files so I needed to rebuild anyway. Having spent hours trying to get .sh working I wanted a complete build using .bat.

jpiesing commented 4 months ago

Is there any change resulting from this issue? I suggest changing as follows.

Replace "Linux / macOS / WSL2:" with "Linux / macOS / Windows without Docker Desktop":

Replace "Windows:" with "Windows with Docker Desktop".

@gitwjr Would this have helped?

gitwjr commented 4 months ago

I would characterize my setup as "windows with wsl 2 and docker desktop". If docker desktop assumes wsl2 then I'd be explicit. If not I'd say "windows (with or without wsl2) with docker desktop". There are other places where it says "windows (without wsl).. " but no mention of wsl2 which left me wondering what I should do.

mlasak commented 4 months ago

There are two options from my point of view to set up the test runner:

  1. You remain in windows world and use docker for desktop and .bat files (even if docker for desktop uses WSL, it should not be used for any manual steps)
  2. You use WSL/WSL2 and install docker inside inside WSL VM (such as Ubuntu) and use .sh files

A mix of WSL steps and docker for desktop ("windows with wsl 2 and docker desktop") is not a official way of setting up and might result in problems. If this is described somewhere then i suggest to edit it and warn users from this. I am referencing your title "./import.sh fails using WSL/Powershell" -> import.sh fails using Powershell is expected -> import.sh fails using WSL (which means inside a linux environment running in VM on your Windows host) is NOT expected

@jpiesing @FritzHeiden pls correct me, if i am wrong with that view.

gitwjr commented 4 months ago

All of my work has been using docker for desktop since I am a small company and do not have to pay for a license to use it. I've never tried docker in my windows/wsl2 computer. That's why the instructions were confusing to me since I used the linux/macOS/windows with wsl setup assuming it was good for docker desktop with wsl. I suggest additional clarification. Currently instructions state "on Linux (applies also for macOS and Windows with WSL2): docker docker-compose git on Windows (without WSL2): Docker-Desktop To access docker the user requires admin permissions (or a special configuration which is out of scope for this guide)"

jpiesing commented 4 months ago

The thing is Docker Desktop includes WSL2 so the choice is Docker Desktop with WSL2 or WSL2 without Docker Desktop.

mlasak commented 4 months ago

Yes, either you decide for "Docker Desktop" or against it.

"Docker Desktop" can use WSL2 as backend, however there is an option which is not WSL2 for non "Home" versions of Windows. I think this is a tech detail which is irrelevant, as in case of "Docker Desktop" you don't "log in" during the setup process into WSL to execute the documented commands.

Happy to get README.md changed from

Linux / macOS / WSL2:

./build.sh master latest

Windows:

.\build.bat master latest

to 

Linux / macOS / inside WSL2 usually Ubuntu (in shell like bash/zh/zsh):

./build.sh master latest

Windows (in cmd.exe or powershell):

.\build.bat master latest
FritzHeiden commented 4 months ago

I'd suggest to explain the two possible setups for windows in more detail under requirements and properly refer to them above the commands.

gitwjr commented 4 months ago

@FritzHeiden I agree more explanation is required. From @mlasak text I would still have questions. Specifically:

FritzHeiden commented 4 months ago

@jpiesing @gitwjr I opened a pull request with some changes to the readme to make the difference between the two possible setups under Windows more clear: https://github.com/cta-wave/dpctf-deploy/pull/89 Human readable version: https://github.com/cta-wave/dpctf-deploy/blob/docs-windows-setup/README.md#windows-setup

Any feedback is welcome

mlasak commented 4 months ago

One option could be to separate instructions more strictly, either explicitly into a Windows only section and Linux only section or even two files. This could help to reduce confusion?

@gitwjr with inside WSL i meant you type in cmd.exe -> "wsl" and land in some linux based environment. Sorry for my bad explanation

jpiesing commented 4 months ago

@jpiesing @gitwjr I opened a pull request with some changes to the readme to make the difference between the two possible setups under Windows more clear: #89 Human readable version: https://github.com/cta-wave/dpctf-deploy/blob/docs-windows-setup/README.md#windows-setup

Any feedback is welcome

I've reviewed that and agree it more clearly spells out the difference between the two windows options. My only comment is that it doesn't mention the port forwarding issues so Docker in Ubuntu in WSL2 can be contacted by a TV set. Also, in this configuration, what goes in host_override and/or in the DNS server that the domain points to?

One option could be to separate instructions more strictly, either explicitly into a Windows only section and Linux only section or even two files. This could help to reduce confusion?

I'm not sure you can separate them easily as WSL2 without Docker Desktop is the same as Linux.

gitwjr commented 4 months ago

@FritzHeiden The new instructions look very clear to me now. As my "Smart" TV (~6 years old) is unable to upgrade the browser to a newer version I cannot run the test on it. Any time I have tried I get a message to to upgrade to newer version but the upgrade fails - my guess is inadequate memory space. Therefore I can't comment on the port forwarding for TVs.

@mlasak Thanks for the response.