search

plasma-umass / scalene

Scalene: a high-performance, high-precision CPU, GPU, and memory profiler for Python with AI-powered optimization proposals
Apache License 2.0
11.57k stars 388 forks source link

[WinError 123] The filename, directory name, or volume label syntax is incorrect #599

Open dave-cz opened 1 year ago

dave-cz commented 1 year ago

I want to profile a code contains APScheduler library but just import it leads to this error:

C:\Users\user\Desktop>scalene tuning.py
Error in program being profiled:
 [WinError 123] The filename, directory name, or volume label syntax is incorrect: '<frozen importlib._bootstrap>'
Traceback (most recent call last):
  File "C:\Users\user\AppData\Local\Programs\Python\Python38\lib\site-packages\scalene\scalene_profiler.py", line 1835, in profile_code
    exec(code, the_globals, the_locals)
  File "tuning.py", line 1, in <module>
    from apscheduler.schedulers.background import BackgroundScheduler
  File "C:\Users\user\AppData\Local\Programs\Python\Python38\lib\site-packages\apscheduler\__init__.py", line 1, in <module>
    from pkg_resources import get_distribution, DistributionNotFound
  File "C:\Users\user\AppData\Local\Programs\Python\Python38\lib\site-packages\pkg_resources\__init__.py", line 3260, in <module>
    def _initialize_master_working_set():
  File "C:\Users\user\AppData\Local\Programs\Python\Python38\lib\site-packages\pkg_resources\__init__.py", line 3234, in _call_aside
    f(*args, **kwargs)
  File "C:\Users\user\AppData\Local\Programs\Python\Python38\lib\site-packages\pkg_resources\__init__.py", line 3272, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "C:\Users\user\AppData\Local\Programs\Python\Python38\lib\site-packages\pkg_resources\__init__.py", line 572, in _build_master
    ws = cls()
  File "C:\Users\user\AppData\Local\Programs\Python\Python38\lib\site-packages\pkg_resources\__init__.py", line 565, in __init__
    self.add_entry(entry)
  File "C:\Users\user\AppData\Local\Programs\Python\Python38\lib\site-packages\pkg_resources\__init__.py", line 621, in add_entry
    for dist in find_distributions(entry, True):
  File "C:\Users\user\AppData\Local\Programs\Python\Python38\lib\site-packages\pkg_resources\__init__.py", line 2096, in find_on_path
    for dist in factory(fullpath):
  File "C:\Users\user\AppData\Local\Programs\Python\Python38\lib\site-packages\pkg_resources\__init__.py", line 2154, in distributions_from_metadata
    if len(os.listdir(path)) == 0:
  File "C:\Users\user\AppData\Local\Programs\Python\Python38\lib\site-packages\scalene\scalene_profiler.py", line 863, in cpu_signal_handler
    Scalene.compute_frames_to_record(),
  File "C:\Users\user\AppData\Local\Programs\Python\Python38\lib\site-packages\scalene\scalene_profiler.py", line 1236, in compute_frames_to_record
    while not Scalene.should_trace(fname):
  File "C:\Users\user\AppData\Local\Programs\Python\Python38\lib\site-packages\scalene\scalene_profiler.py", line 1606, in should_trace
    resolved_filename = str(pathlib.Path(filename).resolve()).lower()
  File "C:\Users\user\AppData\Local\Programs\Python\Python38\lib\pathlib.py", line 1181, in resolve
    s = self._flavour.resolve(self, strict=strict)
  File "C:\Users\user\AppData\Local\Programs\Python\Python38\lib\pathlib.py", line 206, in resolve
    s = self._ext_to_normal(_getfinalpathname(s))
OSError: [WinError 123] The filename, directory name, or volume label syntax is incorrect: '<frozen importlib._bootstrap>'
Scalene: Program did not run for long enough to profile.
NOTE: The GPU is currently running in a mode that can reduce Scalene's accuracy when reporting GPU utilization.
Run once as Administrator or root (i.e., prefixed with `sudo`) to enable per-process GPU accounting.

tuning.py

from apscheduler.schedulers.background import BackgroundScheduler

def add():
    a = []
    for i in range(10000000):
        a.append(i)
    return a

if __name__ == '__main__':
    add_output = add()
    print(len(add_output))

Without apscheduler import Scalene works well.

Desktop:

magnussommarsjo commented 1 year ago

I got the same type of error. Solved it by downgrading to version 1.5.16

emeryberger commented 1 year ago

Does not reproduce on Windows 11, Python 3.10 or Python 3.11, using Scalene 1.5.21.2. Can you all try the latest repo version? Thanks!

dave-cz commented 1 year ago

Does not reproduce on Windows 11, Python 3.10 or Python 3.11, using Scalene 1.5.21.2. Can you all try the latest repo version? Thanks!

scalene 1.5.21.4 (from PyPi)

C:\Users\user\Desktop>scalene tuning.py
Error in program being profiled:
 [WinError 123] The filename, directory name, or volume label syntax is incorrect: '<frozen importlib._bootstrap>'
Traceback (most recent call last):
  File "C:\Users\user\AppData\Local\Programs\Python\Python38\lib\site-packages\scalene\scalene_profiler.py", line 1836, in profile_code
    exec(code, the_globals, the_locals)
  File "tuning.py", line 1, in <module>
    from apscheduler.schedulers.background import BackgroundScheduler
  File "C:\Users\user\AppData\Local\Programs\Python\Python38\lib\site-packages\apscheduler\__init__.py", line 1, in <module>
    from pkg_resources import get_distribution, DistributionNotFound
  File "C:\Users\user\AppData\Local\Programs\Python\Python38\lib\site-packages\pkg_resources\__init__.py", line 80, in <module>
    from pkg_resources.extern import appdirs
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 657, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 556, in module_from_spec
  File "C:\Users\user\AppData\Local\Programs\Python\Python38\lib\site-packages\pkg_resources\extern\__init__.py", line 52, in create_module
    return self.load_module(spec.name)
  File "C:\Users\user\AppData\Local\Programs\Python\Python38\lib\site-packages\pkg_resources\extern\__init__.py", line 37, in load_module
    __import__(extant)
  File "C:\Users\user\AppData\Local\Programs\Python\Python38\lib\site-packages\pkg_resources\_vendor\appdirs.py", line 560, in <module>
    import win32com.shell
  File "C:\Users\user\AppData\Local\Programs\Python\Python38\lib\site-packages\win32com\__init__.py", line 8, in <module>
    import pythoncom
  File "C:\Users\user\AppData\Local\Programs\Python\Python38\lib\site-packages\pythoncom.py", line 2, in <module>
    import pywintypes
  File "C:\Users\user\AppData\Local\Programs\Python\Python38\lib\site-packages\win32\lib\pywintypes.py", line 126, in <module>
    __import_pywin32_system_module__("pywintypes", globals())
  File "C:\Users\user\AppData\Local\Programs\Python\Python38\lib\site-packages\win32\lib\pywintypes.py", line 52, in __import_pywin32_system_module__
    import _win32sysloader
  File "C:\Users\user\AppData\Local\Programs\Python\Python38\lib\site-packages\scalene\scalene_profiler.py", line 864, in cpu_signal_handler
    Scalene.compute_frames_to_record(),
  File "C:\Users\user\AppData\Local\Programs\Python\Python38\lib\site-packages\scalene\scalene_profiler.py", line 1237, in compute_frames_to_record
    while not Scalene.should_trace(fname):
  File "C:\Users\user\AppData\Local\Programs\Python\Python38\lib\site-packages\scalene\scalene_profiler.py", line 1607, in should_trace
    resolved_filename = str(pathlib.Path(filename).resolve()).lower()
  File "C:\Users\user\AppData\Local\Programs\Python\Python38\lib\pathlib.py", line 1181, in resolve
    s = self._flavour.resolve(self, strict=strict)
  File "C:\Users\user\AppData\Local\Programs\Python\Python38\lib\pathlib.py", line 206, in resolve
    s = self._ext_to_normal(_getfinalpathname(s))
OSError: [WinError 123] The filename, directory name, or volume label syntax is incorrect: '<frozen importlib._bootstrap>'
Scalene: Program did not run for long enough to profile.
NOTE: The GPU is currently running in a mode that can reduce Scalene's accuracy when reporting GPU utilization.
Run once as Administrator or root (i.e., prefixed with `sudo`) to enable per-process GPU accounting.

repo version

C:\Users\user\Desktop>scalene tuning.py
10000000
Scalene: An exception of type FileNotFoundError occurred. Arguments:
(2, 'No such file or directory')
Traceback (most recent call last):
  File "C:\Users\user\AppData\Local\Programs\Python\Python38\lib\site-packages\scalene\scalene_profiler.py", line 2063, in run_profiler
    exit_status = profiler.profile_code(
  File "C:\Users\user\AppData\Local\Programs\Python\Python38\lib\site-packages\scalene\scalene_profiler.py", line 1879, in profile_code
    Scalene.generate_html(
  File "C:\Users\user\AppData\Local\Programs\Python\Python38\lib\site-packages\scalene\scalene_profiler.py", line 1806, in generate_html
    gui_js = gui_file.read_text()
  File "C:\Users\user\AppData\Local\Programs\Python\Python38\lib\pathlib.py", line 1236, in read_text
    with self.open(mode='r', encoding=encoding, errors=errors) as f:
  File "C:\Users\user\AppData\Local\Programs\Python\Python38\lib\pathlib.py", line 1222, in open
    return io.open(self, mode, buffering, encoding, errors, newline,
  File "C:\Users\user\AppData\Local\Programs\Python\Python38\lib\pathlib.py", line 1078, in _opener
    return self._accessor.open(self, flags, mode)
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\user\\AppData\\Local\\Programs\\Python\\Python38\\lib\\site-packages\\scalene\\scalene-gui\\scalene-gui.js'

NOTE: The GPU is currently running in a mode that can reduce Scalene's accuracy when reporting GPU utilization.
Run once as Administrator or root (i.e., prefixed with `sudo`) to enable per-process GPU accounting.

It looks like a problem somewhere else, apply params --cli --json --outfile ... works fine and using APScheduler in code leads to note

Scalene currently only supports the `multiprocessing` library on Mac and Unix platforms.