[Issue]: Filename sanitization error.

brknsoul commented 1 year ago

Issue Description

I save my images in G:\SDoutput, which is a different drive to where SD.Next is located. It seem the file names are being incorrectly sanatized;

09:48:05-039429 DEBUG    Filename sanitize: input="G:\SDoutput\grids\2023-10-21\00000-2023-10-21-2172697532-woman long blue hair red eyes wears a-grid.jpg"    
                         parts=('G:\\', 'SDoutput', 'grids', '2023-10-21', '00000-2023-10-21-2172697532-woman long blue hair red eyes wears a-grid')           
                         output=G__\SDoutput\grids\2023-10-21\00000-2023-10-21-2172697532-woman long blue hair red eyes wears a-grid.jpg                       
09:48:05-041396 DEBUG    Saving: image="G__\SDoutput\grids\2023-10-21\00000-2023-10-21-2172697532-woman long blue hair red eyes wears a-grid.jpg" type=JPEG    
                         size=2048x1536

and are now found in (sdnextfolder)\G__\SDoutput instead.

Version Platform Description

09:41:44-703233 INFO Starting SD.Next
09:41:44-705235 INFO Python 3.10.11 on Windows
09:41:44-996234 INFO Version: app=sd.next updated=2023-10-20 hash=e1c46427 url=https://github.com/vladmandic/automatic/tree/master
09:41:45-727026 INFO Platform: arch=AMD64 cpu=Intel64 Family 6 Model 158 Stepping 11, GenuineIntel system=Windows release=Windows-10-10.0.19045-SP0
python=3.10.11
09:41:45-729032 DEBUG Setting environment tuning
09:41:45-731027 DEBUG Torch overrides: cuda=False rocm=False ipex=False diml=True openvino=False
09:41:45-733027 DEBUG Torch allowed: cuda=False rocm=False ipex=False diml=True openvino=False
09:41:45-740026 INFO Using DirectML Backend

Relevant log output

No response

Backend

Original

Model

SD 1.5

Acknowledgements

[X] I have read the above and searched for existing issues
[X] I confirm that this is classified correctly and its not an extension issue

vladmandic commented 1 year ago

sorry about that. fixed.

cjps-br commented 1 year ago

I'm sorry, but I don't think the change solved the problem. I also save my images in a folder defined in the data_path and they are now no longer being saved there but in the sd.next folder. I tried several combinations in the configuration fields in the "settings/image naming & paths" menu and attached images of the configuration that I left last and also an image with the folders that were created, highlighted.

Captura de tela 2023-10-20 224650

cjps-br commented 1 year ago

complementing: I ran sd.next with debug turned on and captured the output when the file was generated:

22:42:51-750940 DEBUG Filename sanitize: input="/outputs/text\GTM_photoV20/2023-10-20\sdnext-20231020_224251-4074116697.jpg" parts=('\', 'outputs', 'text', 'GTM_photoV20', '2023-10-20', 'sdnext-20231020224251-4074116697') output=\outputs\text\GTM_photoV20\2023-10-20\sdnext-20231020224251-4074116697.jpg 22:42:51-754946 DEBUG Saving: image="\outputs\text\GTM_photoV20\2023-10-20\sdnext-20231020224251-4074116697.jpg" type=JPEG size=512x768 22:42:51-760979 DEBUG Saving: text="\outputs\text\GTM_photoV20\2023-10-20\sdnext-20231020_224251-4074116697.txt"

yoshi244 commented 1 year ago

I'm sorry, but I don't think the change solved the problem. I also save my images in a folder defined in the data_path and they are now no longer being saved there but in the sd.next folder. I tried several combinations in the configuration fields in the "settings/image naming & paths" menu and attached images of the configuration that I left last and also an image with the folders that were created, highlighted.

Want to confirm I too am suffering the same issue as you.

cjps-br commented 1 year ago

@vladmandic, I would like to make a modification suggestion for the sanitize function:

def sanitize(self, filename):

    invalid_chars = '#<>.;"/\\|?*\n\t\r'
    invalid_prefix = ''
    invalid_suffix = '.'
    directory, file_name = os.path.split(filename)  #suggestion
    fn, ext = os.path.splitext(file_name) #suggestion
    parts = Path(fn).parts
    newparts = []
    for part in parts:
        part = part.translate({ord(x): '_' for x in invalid_chars})
        part = part.lstrip(invalid_prefix)
        part = part.rstrip(invalid_suffix)
        newparts.append(part)
    fn = Path(*newparts)
    max_length = max(os.statvfs(__file__).f_namemax if hasattr(os, 'statvfs') else 128, 250)
    fn = str(fn)[:max_length - max(4, len(ext))] + ext
    fn = os.path.join(directory, fn) #suggestion
    shared.log.debug(f'Filename sanitize: input="{filename}" parts={parts} output={fn}')
    return fn

I don't know if it's the "most elegant" solution, or if it will add some problems in other parts of the code, but from the tests I did in my environment, it solves the process of saving the images.

In this modification that I am suggesting, the function only works on the file name and not on the file path.

jeditobe1 commented 1 year ago

It appears that on linux, the leading slash is being treated as a part, is being sanitized, and a slash then inserted after the sanitized part before the first folder name:

Filename sanitize:                                     
2023-10-21 03:41:26                          input="/output/images/2023-10-21/00000-dreamshaper_8-to
2023-10-21 03:41:26                          p quality masterpiece rule of thirds half body.png"    
2023-10-21 03:41:26                          parts=('/', 'output', 'images', '2023-10-21',          
2023-10-21 03:41:26                          '00000-dreamshaper_8-top quality masterpiece rule of   
2023-10-21 03:41:26                          thirds half body')                                     
2023-10-21 03:41:26                          output=_/output/images/2023-10-21/00000-dreamshaper_8-t
2023-10-21 03:41:26                          op quality masterpiece rule of thirds half body.png

xGeekpower commented 12 months ago

oh so this was a thing was wondering why my images weren't saving anymore in the correct folder, and they don't even save all individual pictures if you do a batch it just saves the grid and overwrites pictures as well.

cibernicola commented 12 months ago

Latest version, Windows 10, same problem. It adds a "_" between volume leter and path.

vladmandic commented 12 months ago

i should not have tried a quick fix on friday evening and instead take proper time. and this is in general a bigger issue (historical as well as new problems introduced recently).

i've pretty much refactored that part of the code and should be fixed now.

if there are any further issues, please set env variable SD_PATH_DEBUG=true and run webui --debug to capture all required logs and i'll reopen.

cibernicola commented 12 months ago

17:42:12-549684 DEBUG Logger: file=L:\automatic\sdnext.log level=10 size=0 mode=create 17:42:12-554685 INFO Starting SD.Next 17:42:12-556684 INFO Python 3.10.7 on Windows 17:42:13-071684 INFO Version: app=sd.next updated=2023-10-21 hash=b648acf9

Changing error:

Traceback (most recent call last):
  File "L:\automatic\venv\lib\site-packages\gradio\queueing.py", line 388, in call_prediction
    output = await route_utils.call_process_api(
  File "L:\automatic\venv\lib\site-packages\gradio\route_utils.py", line 219, in call_process_api
    output = await app.get_blocks().process_api(
  File "L:\automatic\venv\lib\site-packages\gradio\blocks.py", line 1440, in process_api
    data = self.postprocess_data(fn_index, result["prediction"], state)
  File "L:\automatic\venv\lib\site-packages\gradio\blocks.py", line 1341, in postprocess_data
    prediction_value = block.postprocess(prediction_value)
  File "L:\automatic\venv\lib\site-packages\gradio\components\gallery.py", line 196, in postprocess
    file = self.pil_to_temp_file(img, dir=self.DEFAULT_TEMP_DIR)
  File "L:\automatic\modules\ui_tempdir.py", line 49, in pil_to_temp_file
    exists = os.path.isfile(already_saved_as)
  File "C:\Python310\lib\genericpath.py", line 30, in isfile
    st = os.stat(path)
TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType

finefin commented 12 months ago

I get the TypeError as well, but only when using ControlNet

vladmandic commented 12 months ago

exists = os.path.isfile(already_saved_as)

fixed that as well.

cibernicola commented 12 months ago

Version: app=sd.next updated=2023-10-21 hash=9a1c52f4 Confirming, after several reboots, working fine.

Tillerz commented 12 months ago

Hi, I just found another one related to this.

If you have "save prompt as text file" (dp_write_prompts_to_file) enabled and click SAVE, you get this error.

10:48:53-521535 ERROR    executing callback: D:\SD\automatic.latest\extensions\sd-dynamic-prompts\sd_dynamic_prompts\callbacks.py before_image_saved_callback:
                         FileNotFoundError
┌───────────────────────────────────────────────────────────── Traceback (most recent call last) ─────────────────────────────────────────────────────────────
│ D:\SD\automatic.latest\modules\script_callbacks.py:215 in before_image_saved_callback
│
│   214 │   │   │   t0 = time.time()
│ > 215 │   │   │   c.callback(params)
│   216 │   │   │   timer(t0, c.script, 'before_image_saved')
│
│ D:\SD\automatic.latest\extensions\sd-dynamic-prompts\sd_dynamic_prompts\callbacks.py:45 in on_save
│
│   44 │   │   prompt_filename = image_name.with_suffix(".csv")
│ > 45 │   │   prompt_writer.write_prompts(prompt_filename)
│   46
│
│ D:\SD\automatic.latest\extensions\sd-dynamic-prompts\sd_dynamic_prompts\prompt_writer.py:51 in write_prompts
│
│   50 │   │   path = Path(path)
│ > 51 │   │   with path.open("w", encoding=constants.DEFAULT_ENCODING, errors="ignore") as f:
│   52 │   │   │   writer = csv.writer(f)
│
│ C:\apps\python3.10\lib\pathlib.py:1119 in open
│
│   1118 │   │   │   encoding = io.text_encoding(encoding)
│ > 1119 │   │   return self._accessor.open(self, mode, buffering, encoding, errors,
│   1120 │   │   │   │   │   │   │   │      newline)
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
FileNotFoundError: [Errno 2] No such file or directory: 'D:\\SD\\images\\outputs\\2023-10-22\\00000-20231022104802-1709831843.csv'

vladmandic commented 12 months ago

@Tillerz look at who is reporting that:

extensions\sd-dynamic-prompts\sd_dynamic_prompts\prompt_writer.py

my guess is that its trying to write a file in a subfolder that is not yet created. i'll move the callback to be executed a bit later so path is likely created, but its really an extension issue.

Tillerz commented 12 months ago

I have the "write to sub folder" thing activated with [date] as the format, so it probably didn't generate that sub folder yet.

vladmandic commented 12 months ago

i'll move folder creation to earlier in function.

vladmandic / automatic