Closed thehumandistraction closed 4 years ago
Actually, it looks like I didn't test this properly enough as there seems to be a larger issue with flac2all as a whole. While initially copy works properly after the change in shell.py, it seems to fail after it goes through a couple of tracks and copyable files:
UTC~20:44:35: We have 1487 flac files to convert
UTC~20:44:35: We have 191 non-flac files to copy across
UTC~20:44:35: Spawning execution process
UTC~20:44:35: Processing: 01-take-on-me.flac target: vorbis
UTC~20:44:35: /media/data-disk/tunes/.directory => ./vorbis
UTC~20:44:36: /media/data-disk/tunes/five-iron-frenzy-between-the-pavement-and-the-stars-wav.zip => ./vorbis
UTC~20:44:36: /media/data-disk/tunes/world-is-square-stay-awhile-and-listen-wav.zip => ./vorbis
UTC~20:44:37: Processing: 06-cry-wolf.flac target: vorbis
UTC~20:44:38: /media/data-disk/tunes/world-of-warcraft-battle-for-azeroth-wav.zip => ./vorbis
UTC~20:44:38: /media/data-disk/tunes/a-ha/the-singles-1984-2004/AlbumArtSmall.jpg => ./vorbis/a-ha/the-singles-1984-2004
UTC~20:44:39: /media/data-disk/tunes/a-ha/the-singles-1984-2004/cover.jpg => ./vorbis/a-ha/the-singles-1984-2004
UTC~20:44:39: /media/data-disk/tunes/a-ha/the-singles-1984-2004/Folder.jpg => ./vorbis/a-ha/the-singles-1984-2004
Traceback (most recent call last):
File "/bin/flac2all", line 11, in <module>
load_entry_point('flac2all==5.2', 'console_scripts', 'flac2all')()
File "/usr/lib/python3.8/site-packages/flac2all_pkg/__init__.py", line 440, in main
threaded_encode()
File "/usr/lib/python3.8/site-packages/flac2all_pkg/multiprocess_encode.py", line 141, in encode
copytarget(srcfile, outdir)
File "/usr/lib/python3.8/shutil.py", line 409, in copy
copyfile(src, dst, follow_symlinks=follow_symlinks)
File "/usr/lib/python3.8/shutil.py", line 259, in copyfile
with open(src, 'rb') as fsrc, open(dst, 'wb') as fdst:
FileNotFoundError: [Errno 2] No such file or directory: './vorbis/brave-saint-saturn/anti-meridian'
UTC~20:44:40: Processing: 11-crying-in-the-rain.flac target: vorbis
Specifically, flac2all seems to get confused and mixes up the filename with the album directory name. https://imgur.com/a/EjDkehk
Greetings, thank you for bringing this to my attention. Can you please provide the exact shell command you ran? Then I will try to replicate on my test set.
Sorry for the late response. Here's the command I use:
flac2all vorbis --copy --vorbis-options='quality=3' /media/data-disk/tunes
Thanks, will try to replicate and have a look. Will update ticket with results.
Bug confirmed on an Arch system, package installed today.
After changing line 45 of shell/.py as suggested by thehumandistraction, non-flac files were properly copied, but a new error message was generated by the log generating function. Bug may be unrelated, though.
My command was:
flac2all mp3 /my/input/dir/ --lame-options='V 0' --threads="$(nproc --ignore=1)" --copy -o /my/output/dir
And the error traceback:
UTC~23:56:09: Processing run log...
Traceback (most recent call last):
File "/usr/bin/flac2all", line 11, in <module>
load_entry_point('flac2all==5.0', 'console_scripts', 'flac2all')()
File "/usr/lib/python3.8/site-packages/flac2all_pkg/__init__.py", line 427, in main
threaded_encode()
File "/usr/lib/python3.8/site-packages/flac2all_pkg/multiprocess_encode.py", line 176, in encode
failures = generate_summary(start_time, end_time, len(files), result_log, opts['outdir'])
TypeError: generate_summary() takes 4 positional arguments but 5 were given
Ok, I have made fixes in branch "issue-45". Can you please test this branch against the previously failing files and report results.
I'd be glad to test if you tell me how to install a test copy of your git repo without messing up my pacman-installed version.
On Thu, Feb 27, 2020 at 9:39 AM ZivaVatra notifications@github.com wrote:
Ok, I have made fixes in branch "issue-45". Can you please test this branch against the previously failing files and report results.
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/ZivaVatra/flac2all/issues/45?email_source=notifications&email_token=AAL5HKZASIA3JKYKSQ77ZP3RE7NEFA5CNFSM4KV3YFIKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENE2AGQ#issuecomment-592027674, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAL5HK3NMHGLUV2PHKRIC2DRE7NEFANCNFSM4KV3YFIA .
--
Stefano Franchi
stefano.franchi@gmail.com stefano@tamu.edu https://www.researchgate.net/profile/Stefano_Franchi https://www.researchgate.net/profile/Stefano_Franchi
You should be able to just check out the branch directly using git. See readme for basic instructions: https://github.com/ZivaVatra/flac2all/blob/master/README.md#Development
Effectively your steps would be as follows:
That should work. Once done testing you can just delete $directory
On Thu, Feb 27, 2020 at 11:21 AM ZivaVatra notifications@github.com wrote:
You should be able to just check out the branch directly using git. See readme for basic instructions: https://github.com/ZivaVatra/flac2all/blob/master/README.md#Development
Effectively your steps would be as follows:
- Check out branch "issue-45" using git into $directory
- cd to $directory/flac2all_pkg 3.run "python3 ./init.py mp3 /my/input/dir/ --lame-options='V 0' --threads="$(nproc --ignore=1)" --copy -o /my/output/dir"
That should work.
Thanks for the quick reply. I tested issue-45 branch on the same folder as yesterday and got no errors this time, including a correct log/report.
Cheers,
Stefano --
Stefano Franchi
stefano.franchi@gmail.com stefano@tamu.edu https://www.researchgate.net/profile/Stefano_Franchi https://www.researchgate.net/profile/Stefano_Franchi
It looks like it's half-fixed. It's now processing the files and folders correctly. However, while the copy function seems to work based on the output:
UTC~21:20:32: Processing: 08-ammodramus.flac target: vorbis
UTC~21:20:32: Processing: 09-fields-of-the-fallen.flac target: vorbis
UTC~21:20:32: Processing: 10-always-just-beneath-the-dawn.flac target: vorbis
UTC~21:20:32: Processing: 11-fortress-of-solitude.flac target: vorbis
UTC~21:20:32: Processing: 12-blessed-are-the-land-mines.flac target: vorbis
UTC~21:20:32: Processing: 13-aegolius.flac target: vorbis
UTC~21:20:32: Processing: 14-begin-again.flac target: vorbis
UTC~21:20:33: Processing: 15-these-frail-hands.flac target: vorbis
UTC~21:20:33: Processing: 16-invictus.flac target: vorbis
UTC~21:20:33: Processing: 17-here-is-the-news-(elo-style-radio-edit).flac target: vorbis
UTC~21:20:33: /media/data-disk/tunes/brave-saint-saturn/anti-meridian/cover.jpg => /home/thomas/vorbis/brave-saint-saturn/anti-meridian
UTC~21:20:33: /media/data-disk/tunes/brave-saint-saturn/light-of-things-hoped-for-the/cover.jpg => /home/thomas/vorbis/brave-saint-saturn/light-of-things-hoped-for-the
The copyable files/folders are not actually being copied to the new output folder.
Yep, confirmed on a subsequent test. I had overlooked the copy function the first time.
S.
On Thu, Feb 27, 2020 at 3:35 PM Thomas Pifer notifications@github.com wrote:
It looks like it's half-fixed. It's now processing the files and folders correctly. However, while the copy function seems to work based on the output:
UTC~21:20:32: Processing: 08-ammodramus.flac target: vorbis UTC~21:20:32: Processing: 09-fields-of-the-fallen.flac target: vorbis UTC~21:20:32: Processing: 10-always-just-beneath-the-dawn.flac target: vorbis UTC~21:20:32: Processing: 11-fortress-of-solitude.flac target: vorbis UTC~21:20:32: Processing: 12-blessed-are-the-land-mines.flac target: vorbis UTC~21:20:32: Processing: 13-aegolius.flac target: vorbis UTC~21:20:32: Processing: 14-begin-again.flac target: vorbis UTC~21:20:33: Processing: 15-these-frail-hands.flac target: vorbis UTC~21:20:33: Processing: 16-invictus.flac target: vorbis UTC~21:20:33: Processing: 17-here-is-the-news-(elo-style-radio-edit).flac target: vorbis UTC~21:20:33: /media/data-disk/tunes/brave-saint-saturn/anti-meridian/cover.jpg => /home/thomas/vorbis/brave-saint-saturn/anti-meridian UTC~21:20:33: /media/data-disk/tunes/brave-saint-saturn/light-of-things-hoped-for-the/cover.jpg => /home/thomas/vorbis/brave-saint-saturn/light-of-things-hoped-for-the
The copyable files/folders are not actually being copied to the new output folder.
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/ZivaVatra/flac2all/issues/45?email_source=notifications&email_token=AAL5HK6HOHUULKMHFBSSPE3RFAWZRA5CNFSM4KV3YFIKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENGBGSI#issuecomment-592188233, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAL5HK3IHT4O7QM4WZJT7GLRFAWZRANCNFSM4KV3YFIA .
--
Stefano Franchi
stefano.franchi@gmail.com stefano@tamu.edu https://www.researchgate.net/profile/Stefano_Franchi https://www.researchgate.net/profile/Stefano_Franchi
Thanks @cleinias . Good to hear it is working for you @thehumandistraction Are the files being copied at all? That is odd as on my local test it copied the files fine. I will re-run the branch now to confirm
I ran issue-45 branch a few times on other dirs---files were never copied
S.
On Fri, Feb 28, 2020 at 4:18 AM ZivaVatra notifications@github.com wrote:
Thanks @cleinias https://github.com/cleinias . Good to hear it is working for you @thehumandistraction https://github.com/thehumandistraction Are the files being copied at all? That is odd as on my local test it copied the files fine. I will re-run the branch now to confirm
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ZivaVatra/flac2all/issues/45?email_source=notifications&email_token=AAL5HKZFNSGS3SAEBRXVS5DRFDQHNA5CNFSM4KV3YFIKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENIAOBI#issuecomment-592447237, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAL5HK37PD5VDNZNMHINOYLRFDQHNANCNFSM4KV3YFIA .
--
Stefano Franchi
stefano.franchi@gmail.com stefano@tamu.edu https://www.researchgate.net/profile/Stefano_Franchi https://www.researchgate.net/profile/Stefano_Franchi
Ok, isolated the issue and fix applied. Please pull latest commits and retry.
Tested on a couple of CDs, works all right now.
Cheers,
S,
On Fri, Feb 28, 2020 at 11:09 AM ZivaVatra notifications@github.com wrote:
Ok, isolated the issue and fix applied. Please pull latest commits and retry.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ZivaVatra/flac2all/issues/45?email_source=notifications&email_token=AAL5HK4GZTX5JBT3GODYWKDRFFAMZA5CNFSM4KV3YFIKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENJIF3I#issuecomment-592610029, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAL5HKYE3VOOHHS7ZPBSKOLRFFAMZANCNFSM4KV3YFIA .
--
Stefano Franchi
stefano.franchi@gmail.com stefano@tamu.edu https://www.researchgate.net/profile/Stefano_Franchi https://www.researchgate.net/profile/Stefano_Franchi
Excellent, thanks for confirming. Will close ticket and prep changes for the v5.2 release.
The copy function did work, but only for the first album under
/media/data-disk/tunes/a-ha/the-singles-1984-2004/
And 3 files in the parent directory:
/media/data-disk/tunes/five-iron-frenzy-between-the-pavement-and-the-stars-wav.zip
/media/data-disk/tunes/world-is-square-stay-awhile-and-listen-wav.zip
/media/data-disk/tunes/world-of-warcraft-battle-for-azeroth-wav.zip
Ok, I will run some more tests and see if I can replicate the issue.
log processing seems broken with branch "issue-45"
-[mhauke@loki]- -[/data/mp3rip/newrip/Sepultura-Roots-1996]-
$ ls -1
'01-Sepultura-Roots-Roots Bloody Roots.flac'
02-Sepultura-Roots-Attitude.flac
03-Sepultura-Roots-Cut‐Throat.flac
04-Sepultura-Roots-Ratamahatta.flac
'05-Sepultura-Roots-Breed Apart.flac'
06-Sepultura-Roots-Straighthate.flac
07-Sepultura-Roots-Spit.flac
08-Sepultura-Roots-Lookaway.flac
09-Sepultura-Roots-Dusted.flac
'10-Sepultura-Roots-Born Stubborn.flac'
11-Sepultura-Roots-Jasco.flac
12-Sepultura-Roots-Itsári.flac
13-Sepultura-Roots-Ambush.flac
'14-Sepultura-Roots-Endangered Species.flac'
15-Sepultura-Roots-Dictatorshit.flac
'16-Sepultura-Roots-Chaos B.C..flac'
Sepultura-Roots-1996.toc
-[mhauke@loki]- -[/data/mp3rip/newrip/Sepultura-Roots-1996]-
$ flac2all mp3 . --lame-options='V2' --copy --outdir .
UTC~12:07:27: We have 16 flac files to convert
UTC~12:07:27: We have 1 non-flac files to copy across
UTC~12:07:27: Processing: 01-Sepultura-Roots-Roots Bloody Roots.flac target: mp3
UTC~12:07:27: Processing: 08-Sepultura-Roots-Lookaway.flac target: mp3
UTC~12:07:27: Processing: 13-Sepultura-Roots-Ambush.flac target: mp3
UTC~12:07:27: Processing: 14-Sepultura-Roots-Endangered Species.flac target: mp3
UTC~12:07:27: Processing: 06-Sepultura-Roots-Straighthate.flac target: mp3
UTC~12:07:27: Processing: 09-Sepultura-Roots-Dusted.flac target: mp3
Warning: No Genre detected, setting to "Unknown"
UTC~12:07:27: Processing: 15-Sepultura-Roots-Dictatorshit.flac target: mp3
UTC~12:07:27: Processing: 04-Sepultura-Roots-Ratamahatta.flac target: mp3
Warning: No Genre detected, setting to "Unknown"
Warning: No Genre detected, setting to "Unknown"
Warning: No Genre detected, setting to "Unknown"
Warning: No Genre detected, setting to "Unknown"
Warning: No Genre detected, setting to "Unknown"
Warning: No Genre detected, setting to "Unknown"
Warning: No Genre detected, setting to "Unknown"
UTC~12:07:28: /data/mp3rip/newrip/Sepultura-Roots-1996/Sepultura-Roots-1996.toc => ./mp3/Sepultura-Roots-1996.toc
UTC~12:07:30: Processing: 16-Sepultura-Roots-Chaos B.C..flac target: mp3
Warning: No Genre detected, setting to "Unknown"
UTC~12:07:35: Processing: 03-Sepultura-Roots-Cut‐Throat.flac target: mp3
Warning: No Genre detected, setting to "Unknown"
UTC~12:07:36: Processing: 12-Sepultura-Roots-Itsári.flac target: mp3
Warning: No Genre detected, setting to "Unknown"
UTC~12:07:36: Processing: 11-Sepultura-Roots-Jasco.flac target: mp3
Warning: No Genre detected, setting to "Unknown"
UTC~12:07:36: Processing: 05-Sepultura-Roots-Breed Apart.flac target: mp3
Warning: No Genre detected, setting to "Unknown"
UTC~12:07:38: Processing: 02-Sepultura-Roots-Attitude.flac target: mp3
Warning: No Genre detected, setting to "Unknown"
UTC~12:07:38: Processing: 10-Sepultura-Roots-Born Stubborn.flac target: mp3
Warning: No Genre detected, setting to "Unknown"
UTC~12:07:38: Processing: 07-Sepultura-Roots-Spit.flac target: mp3
Warning: No Genre detected, setting to "Unknown"
UTC~12:07:59: Processing Complete!
UTC~12:07:59: Waiting for all running process to complete.
UTC~12:07:59: Processing run log...
Traceback (most recent call last):
File "/usr/bin/flac2all", line 11, in <module>
load_entry_point('flac2all==5.1', 'console_scripts', 'flac2all')()
File "/usr/lib/python3.8/site-packages/flac2all_pkg/__init__.py", line 436, in main
threaded_encode()
File "/usr/lib/python3.8/site-packages/flac2all_pkg/multiprocess_encode.py", line 188, in encode
print_summary(*results)
NameError: name 'print_summary' is not defined
I'm finding the issue-45 version works correctly, however, when running an a large directory hierarchy that has just a few new flac files added, if the -c option is used it does the transcoding as needed in a few minutes and then spends hours very slowly recopying the few hundred small non-flac files (or possibly just listing them, I don't know if it actually copies). Without -c it finishes up the expected few minutes time.
Ok, so it seems the copy function does not go through the multiprocess code. It does however, work in the clustered mode. I will look into fixing that.
ok, it seems a lot of the logic errors stem from the old multiprocessing code. The easiest thing to do was to remove it, and make the whole system work on ZMQ internally. If nothing else, I only have one set of bugs to fix this way.
I have created a new branch, called "version5-purezmq", which in my tests seems to copy things correctly, and do so in parallel. Could you give it a test? FYI: You will need ZMQ and its bindings installed for this branch.
With or without -c option it spawns 20 workers (on my 10 core processor with two threads each, so that sounds right) and then lists, in batches of 23-28 or so (not 20!), this sort of thing:
UTC~15:11:09: n:16 West Nirvana~ce979acc-2836-4428-8f1d-86d45a165f67 t:mp3 s:Outfile exists, skipping
then pauses around 5 seconds before the next batch of "skipping" reports, and so on.
With -c there will be the occasional
UTC~15:34:39: Processing: cover.jpg target: _copy
with
UTC~15:34:39: n:cover.jpg~f8f052bc-630e-4a54-a1d7-f6c85a2685b2 t:_copy s:SUCCESS
somewhere in that batch of output along with all the skipped audio files.
That sounds like how it should be working. the UUID at the end of the file name is the worker who executed the task.
The batches are how often the message bus is polled, and not related to the number of workers. The number of workers will not change based on "-c" or not, what will change is the number of tasks (as each "copy" is counted as a task, as well as the encode).
If you are getting "SUCCESS" for the copy task, then you should be seeing your files copied in the correct directories. If not, try a fresh pull into the branch, as I have been refactoring the copy logic in the last 2 hours.
It appears to work but it's taking 45 minutes go through 15,700 flac files even if nothing actually needs transcoding, as if it's managing to only check a few files per second to see that the output file already exists. The -issue45 version not using the -c option is far faster when no files needed transcoding, I just tried and it took 1 min 10 sec. It was only this slow with -c.
I have to say, that is peculiar. I just ran a time test between the version5-purezmq branch, and the version5 branch (that still has the old multiprocess code in it, and issue45 merged in).
The purezmq branch in my case finishes much faster (35 seconds vs 8 minutes for the old code in real time), but looking at CPU time, the new branch took 2.884seconds, and the old one 0.382seconds.
So the old code is faster in CPU time, but on my system it takes a lot longer to complete in real time, which is why for me the zmq version was always more performant. I am not sure why. Sure there is some overhead in setting up the ZMQ channels, but that should not scale with number of files.
I will run two cycles against my entire FLAC collection (22,386 files), and see how the timings compare. Until then, my results below:
version5-purezmq branch:
flac@Athena:~/flac2all/flac2all_pkg$ time python3 ./__init__.py vorbis -c -n -o ../testoutput ../testinput/
[snip]
UTC~23:11:55: Waiting for local worker to terminate
UTC~23:11:55: input: 90, output: 90
UTC~23:11:55: Writing log file (../testoutput/conversion_results.log)
Done!
real 0m35.914s
user 0m2.500s
sys 0m0.384s
Vs
version5-branch:
flac@Athena:~/flac2all/flac2all_pkg$ time python3 ./__init__.py vorbis -c -n -o ../testoutput ../testinput/
[snip]
| Summary ====================================================================
Total files on input: 90
Total files actually processed: 47
--
Execution rate: 52.22%
Files we managed to convert successfully: 47
Files we failed to convert due to errors: 0
--
Conversion error rate: 0.0%
UTC~23:20:25: Writing log file (../testoutput/conversion_results.log)
Done!
real 8m0.732s
user 0m0.700s
sys 0m0.132s
Ok, I think I have found the cause. With the new changes the version5-purezmq branch is about 2x faster than version5 in CPU time.
Please try the latest changes and let me know if you see any improvement.
Super! Now all the messages about files being skipped or copied scroll by in a blur (instead of slow enough to literally be readable) and its done in a minute and a half clock time. Thanks!
It looks like the copy function does seem to be fixed. :smile:
However, there seem to remain a few bugs.
The main one is that after completion of the encoding/copying, the main flac2all command does not seem to properly terminate.
[SNIP]
UTC~01:30:30: n:03-independence-day~cb62a5d4-8e4e-4f6c-aef2-6bf1e9f8a24... t:vorbis s:SUCCESS
UTC~01:30:30: n:04-mercury~cb62a5d4-8e4e-4f6c-aef2-6bf1e9f8a24e t:vorbis s:SUCCESS
UTC~01:30:32: n:06-through-depths-of-twilight~cb62a5d4-8e4e-4f6c-aef2-6... t:vorbis s:SUCCESS
UTC~01:30:32: Worker 2e166786-a386-46c0-bed8-b1351c7bc2c4 terminated (5 running)
UTC~01:30:43: n:12-move-to-memphis~cb62a5d4-8e4e-4f6c-aef2-6bf1e9f8a24e t:vorbis s:SUCCESS
UTC~01:30:43: Worker cb62a5d4-8e4e-4f6c-aef2-6bf1e9f8a24e terminated (4 running)
UTC~01:30:43: Worker dfd86248-e483-4c6a-a544-39de342c1f97 terminated (3 running)
UTC~01:30:44: n:world-of-warcraft-battle-for-azeroth-wav.zip~6d581376-b... t:_copy s:SUCCESS
UTC~01:30:44: Worker 6d581376-b962-4f26-9841-0b1bb45f4382 terminated (2 running)
UTC~01:30:44: Worker 0f02ef34-f4c8-47ef-9cd7-e62857c0b17c terminated (1 running)
UTC~01:34:27: Worker 19edfd4c-d3d0-43e1-9373-92d60b443a9a not responding, clearing from list (0 remaining)
UTC~01:34:27: Terminated
UTC~01:34:27: Waiting for local worker to terminate
UTC~01:34:27: input: 1873, output: 1873
UTC~01:34:27: Writing log file (/media/secondary-disk/temp/test2/conversion_results.log)
Done!
This then causes all of the threads that were spawned to remain open:
[thomas@Ragnell flac2all_pkg]$ ps aux | grep python
thomas 58584 0.0 0.0 171756 13644 ? Sl 21:04 0:00 python ./__init__.py vorbis -c -n --vorbis-options=quality=3 -o /media/secondary-disk/temp/test2 /media/data-disk/tunes
thomas 58587 0.0 0.0 171760 13728 ? Sl 21:04 0:00 python ./__init__.py vorbis -c -n --vorbis-options=quality=3 -o /media/secondary-disk/temp/test2 /media/data-disk/tunes
thomas 58590 0.0 0.0 171712 13716 ? Sl 21:04 0:00 python ./__init__.py vorbis -c -n --vorbis-options=quality=3 -o /media/secondary-disk/temp/test2 /media/data-disk/tunes
thomas 58593 0.0 0.0 171712 13648 ? Sl 21:04 0:00 python ./__init__.py vorbis -c -n --vorbis-options=quality=3 -o /media/secondary-disk/temp/test2 /media/data-disk/tunes
thomas 58596 0.0 0.0 171712 13652 ? Sl 21:04 0:00 python ./__init__.py vorbis -c -n --vorbis-options=quality=3 -o /media/secondary-disk/temp/test2 /media/data-disk/tunes
thomas 58599 0.0 0.0 171712 13664 ? Sl 21:04 0:00 python ./__init__.py vorbis -c -n --vorbis-options=quality=3 -o /media/secondary-disk/temp/test2 /media/data-disk/tunes
thomas 58602 0.0 0.0 171712 13676 ? Sl 21:04 0:00 python ./__init__.py vorbis -c -n --vorbis-options=quality=3 -o /media/secondary-disk/temp/test2 /media/data-disk/tunes
thomas 58605 0.0 0.0 171712 13688 ? Sl 21:04 0:00 python ./__init__.py vorbis -c -n --vorbis-options=quality=3 -o /media/secondary-disk/temp/test2 /media/data-disk/tunes
thomas 58608 0.0 0.0 171712 13652 ? Sl 21:04 0:00 python ./__init__.py vorbis -c -n --vorbis-options=quality=3 -o /media/secondary-disk/temp/test2 /media/data-disk/tunes
thomas 58611 0.0 0.0 171712 13720 ? Sl 21:04 0:00 python ./__init__.py vorbis -c -n --vorbis-options=quality=3 -o /media/secondary-disk/temp/test2 /media/data-disk/tunes
thomas 58614 0.0 0.0 171712 13724 ? Sl 21:04 0:00 python ./__init__.py vorbis -c -n --vorbis-options=quality=3 -o /media/secondary-disk/temp/test2 /media/data-disk/tunes
thomas 58617 0.0 0.0 171712 13688 ? Sl 21:04 0:00 python ./__init__.py vorbis -c -n --vorbis-options=quality=3 -o /media/secondary-disk/temp/test2 /media/data-disk/tunes
thomas 58620 0.0 0.0 171712 13700 ? Sl 21:04 0:00 python ./__init__.py vorbis -c -n --vorbis-options=quality=3 -o /media/secondary-disk/temp/test2 /media/data-disk/tunes
Because of this the ZMQ ports also remain open causing flac2all to not be able to run again unless all of the previously spawned processes are killed:
UTC~01:15:40: Parent path /media/secondary-disk/temp/test2 does not exist! quitting...
Traceback (most recent call last):
File "./__init__.py", line 442, in <module>
main()
File "./__init__.py", line 435, in main
clustered_encode(localworkers=True)
File "./__init__.py", line 100, in clustered_encode
tsock.bind("tcp://*:2019")
File "zmq/backend/cython/socket.pyx", line 550, in zmq.backend.cython.socket.Socket.bind
File "zmq/backend/cython/checkrc.pxd", line 26, in zmq.backend.cython.checkrc._check_rc
zmq.error.ZMQError: Address already in use
flac2all also seems to be ignoring the directory output test logic. It will always display the error at the beginning:
UTC~01:15:40: Parent path /media/secondary-disk/temp/test2 does not exist! quitting...
and will always continue to run whether or not the path specified does or does not exist and whether or not -n
is passed or not passed.
For reference, the command I'm using is:
python ./__init__.py vorbis -c --vorbis-options='quality=3' -o /media/secondary-disk/temp/test2 /media/data-disk/tunes
Or in testing, adding -n
:
python ./__init__.py vorbis -c --vorbis-options='quality=3' -n -o /media/secondary-disk/temp/test2 /media/data-disk/tunes
I've not seen the ZMQ "address in use" error unless I kill the process uncleanly. I've been running it in a loop and it always terminated just fine. What OS are you running on? I will have to try and replicate that bug first, then I can look at fixing it.
And "-n" not working has been raised in #50 , so I will most likely post updates to that over there when I get to it.
Well, I've been running it in a loop since my last message. It has done 31,204 cycles with no errors or orphaned processes at all. Sorry but so far I am unable to reproduce the issue you are seeing.
I saw it much more with the previous versions, I can confirm I've seen this latest take minutes to terminate after finishing and then leave the hanging processes once but I can't reproduce it.
Yes, the previous versions did have a bug which caused orphaned processes, that was fixed roughly when I fixed the copy code.
One off events like that are hard to debug. If it was a reproducible bug, then we could investigate further and issue a fix. I know the propagation of terminate signals needs to be improved, and that is on my TODO list.
Unless anyone objects or has anything further to add, I would like to close this ticket now? For the "-n" not working, #50 is what you want to watch.
Unless anyone objects or has anything further to add, I would like to close this ticket now? For the "-n" not working, #50 is what you want to watch.
Yes, I agree. Thanks again for taking the time to fix this! :+1:
You are welcome. Closing ticket now.
Greetings,
In the latest version 5.1 the copy function is broken due to
string
being called rather thanstr
in shell.py. This causes flac2all to hang once all music files have been processed. This happens the first time it comes across a file it intends to copy.The specific error can be seen when trying to copy cover art as it processes it's first album:
Changing line 45 from:
if (string.find(os.path.split(indir)[0], dirpath) != -1):
to:
if (str.find(os.path.split(indir)[0], dirpath) != -1):
fixes flac2all and copy functions as it should.
This was tested on an up-to-date Arch Linux system with the latest package from AUR.