search

Ceiridge / WinFsp-MemFs-Extended

tmpfs for Windows / Fully dynamic RAM disk by extending WinFsp's memfs to be variable size and maximally resource efficient without compromising performance
GNU General Public License v3.0
51 stars 2 forks source link

The volume does not contain a recognized file system #2

Closed kutu closed 1 year ago

kutu commented 1 year ago

When trying to save video streams on ramdisk, i get this error

R:\
$ streamlink.exe https://www.youtube.com/watch?v=TjAa0wOe5k4 best -fo test.mp4
[cli][info] Found matching plugin youtube for URL https://www.youtube.com/watch?v=TjAa0wOe5k4
[cli][info] Available streams: audio_mp4a, audio_opus, 144p (worst), 240p, 360p, 480p, 720p, 1080p60 (best)
[cli][info] Opening stream: 1080p60 (muxed-stream)
Traceback (most recent call last):
  File "C:\Users\kutu\AppData\Local\Programs\Python\Python310\lib\ntpath.py", line 660, in realpath
    path = _getfinalpathname(path)
FileNotFoundError: [WinError 2] The system cannot find the file specified: 'R:\\test.mp4'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\kutu\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\kutu\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\kutu\AppData\Local\Programs\Python\Python310\Scripts\streamlink.exe\__main__.py", line 7, in <module>
  File "C:\Users\kutu\AppData\Local\Programs\Python\Python310\lib\site-packages\streamlink_cli\main.py", line 1101, in main
    handle_url()
  File "C:\Users\kutu\AppData\Local\Programs\Python\Python310\lib\site-packages\streamlink_cli\main.py", line 616, in handle_url
    handle_stream(plugin, streams, stream_name)
  File "C:\Users\kutu\AppData\Local\Programs\Python\Python310\lib\site-packages\streamlink_cli\main.py", line 484, in handle_stream
    success = output_stream(stream, formatter)
  File "C:\Users\kutu\AppData\Local\Programs\Python\Python310\lib\site-packages\streamlink_cli\main.py", line 339, in output_stream
    output = create_output(formatter)
  File "C:\Users\kutu\AppData\Local\Programs\Python\Python310\lib\site-packages\streamlink_cli\main.py", line 110, in create_output
    return check_file_output(formatter.path(args.output, args.fs_safe_rules), args.force)
  File "C:\Users\kutu\AppData\Local\Programs\Python\Python310\lib\site-packages\streamlink_cli\main.py", line 76, in check_file_output
    log.info(f"Writing output to\n{path.resolve()}")
  File "C:\Users\kutu\AppData\Local\Programs\Python\Python310\lib\pathlib.py", line 1077, in resolve
    s = self._accessor.realpath(self, strict=strict)
  File "C:\Users\kutu\AppData\Local\Programs\Python\Python310\lib\ntpath.py", line 666, in realpath
    path = _getfinalpathname_nonstrict(path)
  File "C:\Users\kutu\AppData\Local\Programs\Python\Python310\lib\ntpath.py", line 614, in _getfinalpathname_nonstrict
    path = _getfinalpathname(path)
OSError: [WinError 1005] The volume does not contain a recognized file system.
Please make sure that all required file system drivers are loaded and that the volume is not corrupted: 'R:\\'

same command on usual hdd

H:\
$ streamlink.exe https://www.youtube.com/watch?v=TjAa0wOe5k4 best -fo test.mp4
[cli][info] Found matching plugin youtube for URL https://www.youtube.com/watch?v=TjAa0wOe5k4
[cli][info] Available streams: audio_mp4a, audio_opus, 144p (worst), 240p, 360p, 480p, 720p, 1080p60 (best)
[cli][info] Opening stream: 1080p60 (muxed-stream)
[cli][info] Writing output to
H:\test.mp4
[utils.named_pipe][info] Creating pipe streamlinkpipe-12772-1-7950
[utils.named_pipe][info] Creating pipe streamlinkpipe-12772-2-1755
[download][test.mp4] Written 10.6 MB (2m35s @ 84.7 KB/s)
[cli][info] Stream ended
[cli][info] Closing currently open stream...

This is simplest example i found to reproduce this error. I also use your program to compile apps with msbuild, when i try to compile aseprite, msbuild step goes fine, but then ninja fails, and i think it is the same error as with saving video streams.

this is ninja failed compile output

R:\1\aseprite\build
$ ninja aseprite
[1/1500] Building C object third_party\CMakeFiles\tinyexpr.dir\tinyexpr\tinyexpr.c.obj
R:\1\aseprite\third_party\tinyexpr\tinyexpr.c(462): warning C4090: 'initializing': different 'const' qualifiers
R:\1\aseprite\third_party\tinyexpr\tinyexpr.c(479): warning C4090: 'initializing': different 'const' qualifiers
R:\1\aseprite\third_party\tinyexpr\tinyexpr.c(494): warning C4090: 'initializing': different 'const' qualifiers
[106/1500] Generating scripts/symbols.out
Microsoft (R) C/C++ Optimizing Compiler Version 19.33.31629 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

symbols.c
[112/1500] Generating pnglibconf.out
Microsoft (R) C/C++ Optimizing Compiler Version 19.33.31629 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

pnglibconf.c
[117/1500] Generating scripts/prefix.out
Microsoft (R) C/C++ Optimizing Compiler Version 19.33.31629 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

prefix.c
[118/1500] Generating scripts/vers.out
Microsoft (R) C/C++ Optimizing Compiler Version 19.33.31629 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

vers.c
[119/1500] Generating scripts/sym.out
Microsoft (R) C/C++ Optimizing Compiler Version 19.33.31629 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

sym.c
[124/1500] Building C object third_party\curl\lib\CMakeFiles\libcurl.dir\base64.c.obj
FAILED: third_party/curl/lib/CMakeFiles/libcurl.dir/base64.c.obj
C:\PROGRA~2\MICROS~2\2022\BUILDT~1\VC\Tools\MSVC\1433~1.316\bin\Hostx64\x64\cl.exe  /nologo -DBUILDING_LIBCURL -DCMARK_STATIC_DEFINE -DCURL_HIDDEN_SYMBOLS -DHAVE_CONFIG_H -DNDEBUG -DPNG_NO_MMX_CODE -D_CRT_NONSTDC_NO_DEPRECATE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_SECURE_NO_WARNINGS -D_USRDLL -D_WIN32_WINNT=0x0600 -IR:\1\aseprite\third_party\zlib -IR:\1\aseprite\build\third_party\zlib -IR:\1\aseprite\third_party\libpng -IR:\1\aseprite\build\third_party\libpng -IR:\1\aseprite\third_party\tinyxml -IR:\1\aseprite\third_party\pixman\pixman -IR:\1\aseprite\build -IR:\1\aseprite\third_party\giflib -IR:\1\aseprite\third_party\jpeg -IR:\1\aseprite\third_party\curl\include -IR:\1\aseprite\third_party\simpleini -IR:\1\aseprite\third_party\. -IR:\1\aseprite\build\third_party\curl\lib\..\include -IR:\1\aseprite\third_party\curl\lib\.. -IR:\1\aseprite\third_party\curl\lib\..\include -IR:\1\aseprite\build\third_party\curl\lib\.. -IR:\1\aseprite\third_party\curl\lib -IR:\1\aseprite\build\third_party\curl\lib /DWIN32 /D_WINDOWS /W4 /MP /MT /Zi /O2 /Ob1 /D NDEBUG -MT /showIncludes /Fothird_party\curl\lib\CMakeFiles\libcurl.dir\base64.c.obj /Fdthird_party\curl\lib\CMakeFiles\libcurl.dir\libcurl.pdb /FS -c R:\1\aseprite\third_party\curl\lib\base64.c
R:\1\aseprite\third_party\curl\lib\curl_setup.h(247): fatal error C1083: Cannot open include file: 'setup-win32.h': No such file or directory
[129/1500] Building C object third_party\curl\lib\CMakeFiles\libcurl.dir\bufref.c.obj
FAILED: third_party/curl/lib/CMakeFiles/libcurl.dir/bufref.c.obj
C:\PROGRA~2\MICROS~2\2022\BUILDT~1\VC\Tools\MSVC\1433~1.316\bin\Hostx64\x64\cl.exe  /nologo -DBUILDING_LIBCURL -DCMARK_STATIC_DEFINE -DCURL_HIDDEN_SYMBOLS -DHAVE_CONFIG_H -DNDEBUG -DPNG_NO_MMX_CODE -D_CRT_NONSTDC_NO_DEPRECATE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_SECURE_NO_WARNINGS -D_USRDLL -D_WIN32_WINNT=0x0600 -IR:\1\aseprite\third_party\zlib -IR:\1\aseprite\build\third_party\zlib -IR:\1\aseprite\third_party\libpng -IR:\1\aseprite\build\third_party\libpng -IR:\1\aseprite\third_party\tinyxml -IR:\1\aseprite\third_party\pixman\pixman -IR:\1\aseprite\build -IR:\1\aseprite\third_party\giflib -IR:\1\aseprite\third_party\jpeg -IR:\1\aseprite\third_party\curl\include -IR:\1\aseprite\third_party\simpleini -IR:\1\aseprite\third_party\. -IR:\1\aseprite\build\third_party\curl\lib\..\include -IR:\1\aseprite\third_party\curl\lib\.. -IR:\1\aseprite\third_party\curl\lib\..\include -IR:\1\aseprite\build\third_party\curl\lib\.. -IR:\1\aseprite\third_party\curl\lib -IR:\1\aseprite\build\third_party\curl\lib /DWIN32 /D_WINDOWS /W4 /MP /MT /Zi /O2 /Ob1 /D NDEBUG -MT /showIncludes /Fothird_party\curl\lib\CMakeFiles\libcurl.dir\bufref.c.obj /Fdthird_party\curl\lib\CMakeFiles\libcurl.dir\libcurl.pdb /FS -c R:\1\aseprite\third_party\curl\lib\bufref.c
R:\1\aseprite\third_party\curl\lib\urldata.h(131): fatal error C1083: Cannot open include file: 'pop3.h': No such file or directory
[132/1500] Generating scripts/intprefix.out
Microsoft (R) C/C++ Optimizing Compiler Version 19.33.31629 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

intprefix.c
[137/1500] Building CXX object third_party\tinyxml\CMakeFiles\tinyxml.dir\tinyxmlparser.cpp.obj
ninja: build stopped: subcommand failed.

at the end of FAILED block you can see Cannot open include file: 'setup-win32.h': No such file or directory

but file exists R:\1\aseprite\third_party\curl\lib\setup-win32.h

when i do the same steps to compile aseprite on hdd, everything is working as expected

i launch memfs as "Recommended Installation" from your project description

Ceiridge commented 1 year ago

This might be a problem with WinFsp: https://github.com/winfsp/winfsp/issues/427#issuecomment-1145880940 You could try changing the arguments of the launchctl task to mount the drive on \\.\R: for instance and see if it works.

kutu commented 1 year ago

thanks for quick reply, that did solve streamlink issue

ninja failed with different file that cannot be found, and on 2nd attempt to compile aseprite, cmd hanged

this is not first time hang happened, but currently i dont have any reproduceable steps to provide

then, i launched memefs with debugging flags "C:\Program Files (x86)\WinFsp\bin\memefs-x64.exe" -i -F NTFS -l Ram -u "" -m "\\.\R:" -d -1 -D h:\memefs.log

ninja gives different error on each run

R:\1\aseprite\build
$ ninja aseprite
[1/1500] Building C object third_party\CMakeFiles\tinyexpr.dir\tinyexpr\tinyexpr.c.obj
R:\1\aseprite\third_party\tinyexpr\tinyexpr.c(462): warning C4090: 'initializing': different 'const' qualifiers
R:\1\aseprite\third_party\tinyexpr\tinyexpr.c(479): warning C4090: 'initializing': different 'const' qualifiers
R:\1\aseprite\third_party\tinyexpr\tinyexpr.c(494): warning C4090: 'initializing': different 'const' qualifiers
[65/1500] Building C object third_party\jpeg\CMakeFiles\jpeg.dir\jdmaster.c.obj
ninja: build stopped: GetFileAttributesEx(third_party/jpeg/CMakeFiles/jpeg.dir/jdmaster.c.obj): The filename, directory name, or volume label syntax is incorrect.
.

R:\1\aseprite\build
$ ninja aseprite
[39/1436] Generating scripts/symbols.out
Microsoft (R) C/C++ Optimizing Compiler Version 19.33.31629 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

symbols.c
[46/1436] Generating pnglibconf.out
FAILED: third_party/libpng/pnglibconf.out R:/1/aseprite/build/third_party/libpng/pnglibconf.out
cmd.exe /C "cd /D R:\1\aseprite\build\third_party\libpng && D:\programs\cmake\bin\cmake.exe -DINPUT=R:/1/aseprite/build/third_party/libpng/pnglibconf.c -DOUTPUT=R:/1/aseprite/build/third_party/libpng/pnglibconf.out -P R:/1/aseprite/build/third_party/libpng/scripts/genout.cmake"
Microsoft (R) C/C++ Optimizing Compiler Version 19.33.31629 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

pnglibconf.c
awk: error: can't open source file `R:/1/aseprite/third_party/libpng/scripts/dfn.awk' for reading (No such file or directory)
CMake Error at scripts/genout.cmake:86 (message):
  Failed to generate
  R:/1/aseprite/build/third_party/libpng/pnglibconf.out.tf2

[59/1436] Building CXX object third_party\tinyxml\CMakeFiles\tinyxml.dir\tinyxmlerror.cpp.obj
ninja: build stopped: subcommand failed.

R:\1\aseprite\build
$ ninja aseprite
[3/1378] Generating pnglibconf.out
Microsoft (R) C/C++ Optimizing Compiler Version 19.33.31629 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

pnglibconf.c
[5/1378] Generating scripts/vers.out
Microsoft (R) C/C++ Optimizing Compiler Version 19.33.31629 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

vers.c
[9/1378] Generating scripts/sym.out
Microsoft (R) C/C++ Optimizing Compiler Version 19.33.31629 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

sym.c
[11/1378] Generating scripts/prefix.out
Microsoft (R) C/C++ Optimizing Compiler Version 19.33.31629 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

prefix.c
[16/1378] Generating scripts/intprefix.out
Microsoft (R) C/C++ Optimizing Compiler Version 19.33.31629 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

intprefix.c
[27/1378] Building C object third_party\curl\lib\CMakeFiles\libcurl.dir\curl_ntlm_core.c.obj
FAILED: third_party/curl/lib/CMakeFiles/libcurl.dir/curl_ntlm_core.c.obj
C:\PROGRA~2\MICROS~2\2022\BUILDT~1\VC\Tools\MSVC\1433~1.316\bin\Hostx64\x64\cl.exe  /nologo -DBUILDING_LIBCURL -DCMARK_STATIC_DEFINE -DCURL_HIDDEN_SYMBOLS -DHAVE_CONFIG_H -DNDEBUG -DPNG_NO_MMX_CODE -D_CRT_NONSTDC_NO_DEPRECATE -D_CRT_SECURE_NO_DEPRECATE -D_CRT_SECURE_NO_WARNINGS -D_USRDLL -D_WIN32_WINNT=0x0600 -IR:\1\aseprite\third_party\zlib -IR:\1\aseprite\build\third_party\zlib -IR:\1\aseprite\third_party\libpng -IR:\1\aseprite\build\third_party\libpng -IR:\1\aseprite\third_party\tinyxml -IR:\1\aseprite\third_party\pixman\pixman -IR:\1\aseprite\build -IR:\1\aseprite\third_party\giflib -IR:\1\aseprite\third_party\jpeg -IR:\1\aseprite\third_party\curl\include -IR:\1\aseprite\third_party\simpleini -IR:\1\aseprite\third_party\. -IR:\1\aseprite\build\third_party\curl\lib\..\include -IR:\1\aseprite\third_party\curl\lib\.. -IR:\1\aseprite\third_party\curl\lib\..\include -IR:\1\aseprite\build\third_party\curl\lib\.. -IR:\1\aseprite\third_party\curl\lib -IR:\1\aseprite\build\third_party\curl\lib /DWIN32 /D_WINDOWS /W4 /MP /MT /Zi /O2 /Ob1 /D NDEBUG -MT /showIncludes /Fothird_party\curl\lib\CMakeFiles\libcurl.dir\curl_ntlm_core.c.obj /Fdthird_party\curl\lib\CMakeFiles\libcurl.dir\libcurl.pdb /FS -c R:\1\aseprite\third_party\curl\lib\curl_ntlm_core.c
R:\1\aseprite\third_party\curl\lib\curl_setup_once.h(324): fatal error C1083: Cannot open include file: 'curl_ctype.h': No such file or directory
[40/1378] Building C object third_party\libpng\CMakeFiles\png_static.dir\pngrtran.c.obj
ninja: build stopped: subcommand failed.

log file

Ceiridge commented 1 year ago

This might be because of concurrency issues with memefs, I assume. When building with ninja, can you make sure that it only uses one thread in total (maybe there are some command line arguments for that)? Does building with ninja work on a normal hard drive?

kutu commented 1 year ago

Does building with ninja work on a normal hard drive?

yes

you were right, i limited ninja to only run one job at a time, and compilation succeeded, but that eliminates the whole benefit of running it in ram

thanks for your help, i will keep using hdd for parallel compilations like this

please add \\.\R: hint to readme, and maybe mention about concurrency file access limitation

Ceiridge commented 1 year ago

The reworked version that will be released soon does not have this problem anymore.

kutu commented 1 year ago

if you mean fixing concurrency issue, i can do test on my pc, if you provide memefs-x64.exe test file following question, do i need to use newest winfsp, i'm currently on 1.11.x

Ceiridge commented 1 year ago

I will still need to fix many things before I can release it. You will also need the latest WinFsp version, whereas updating requires uninstalling with restarting your computer and then installing the latest version.

Ceiridge commented 1 year ago

@kutu You can now try it out if you want to https://github.com/Ceiridge/WinFsp-MemFs-Extended/releases

kutu commented 1 year ago

worked with first try aseprite compiled within 1min thanks for this addition