search

klay-music / klay-beam

Our Apache Beam Transforms and Pipelines
1 stars 0 forks source link

local variable `sr` referenced before assignment in `LoadAudio` when running `job_rms` #69

Closed cyrusvahidi closed 11 months ago

cyrusvahidi commented 11 months ago

Overview

I am running a job to extract RMS from 200K files. Around 40K files, I hit this error, perhaps due to inability to read the audio file. It looks like we return empty array and sr when librosa throws a RuntimeError. However, we're trying to reference the variable sr which has not been set.

Fix

I suggest that we explicity assign sr = None before returns the null tuple.

https://github.com/klay-music/klay-beam/blob/bc8e8ddf11dc6e2d0e87bd1d98c5c25efece7ece/klay_beam/src/klay_beam/transforms.py#L345

Stack Trace

INFO:root:Writing to: /data/scratch/acw532/all/26917_10.mp3.rms_50hz.npy
INFO:root:Found 0 of: /data/scratch/acw532/all/79171_B2.rms_50hz.npy
INFO:root:Loading: /data/scratch/acw532/all/79171_B2.mp3
WARNING:root:Error loading audio: /data/scratch/acw532/all/79171_B2.mp3
ERROR:apache_beam.runners.common:local variable 'sr' referenced before assignment [while running 'LoadAudio']
Traceback (most recent call last):
  File "/data/home/acw532/.conda/envs/rms-dev/lib/python3.10/site-packages/klay_beam/transforms.py", line 333, in process
    audio_array, sr = librosa.load(file_like, sr=self.target_sr, mono=self.mono)
  File "/data/home/acw532/.conda/envs/rms-dev/lib/python3.10/site-packages/librosa/core/audio.py", line 185, in load
    raise exc
  File "/data/home/acw532/.conda/envs/rms-dev/lib/python3.10/site-packages/librosa/core/audio.py", line 175, in load
    y, sr_native = __soundfile_load(path, offset, duration, dtype)
  File "/data/home/acw532/.conda/envs/rms-dev/lib/python3.10/site-packages/librosa/core/audio.py", line 208, in __soundfile_load
    context = sf.SoundFile(path)
  File "/data/home/acw532/.conda/envs/rms-dev/lib/python3.10/site-packages/soundfile.py", line 658, in __init__
    self._file = self._open(file, mode_int, closefd)
  File "/data/home/acw532/.conda/envs/rms-dev/lib/python3.10/site-packages/soundfile.py", line 1216, in _open
    raise LibsndfileError(err, prefix="Error opening {0!r}: ".format(self.name))
soundfile.LibsndfileError: Error opening <_io.BufferedReader name='/data/scratch/acw532/hardwaxall/79171_B2.mp3'>: Format not recognised.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "apache_beam/runners/common.py", line 1435, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 637, in apache_beam.runners.common.SimpleInvoker.invoke_process
  File "/data/home/acw532/.conda/envs/rms-dev/lib/python3.10/site-packages/klay_beam/transforms.py", line 345, in process
    return (readable_file.metadata.path, np.ndarray([]), sr)
UnboundLocalError: local variable 'sr' referenced before assignment
Traceback (most recent call last):
  File "/data/home/acw532/.conda/envs/rms-dev/lib/python3.10/site-packages/klay_beam/transforms.py", line 333, in process
    audio_array, sr = librosa.load(file_like, sr=self.target_sr, mono=self.mono)
  File "/data/home/acw532/.conda/envs/rms-dev/lib/python3.10/site-packages/librosa/core/audio.py", line 185, in load
    raise exc
  File "/data/home/acw532/.conda/envs/rms-dev/lib/python3.10/site-packages/librosa/core/audio.py", line 175, in load
    y, sr_native = __soundfile_load(path, offset, duration, dtype)
  File "/data/home/acw532/.conda/envs/rms-dev/lib/python3.10/site-packages/librosa/core/audio.py", line 208, in __soundfile_load
    context = sf.SoundFile(path)
  File "/data/home/acw532/.conda/envs/rms-dev/lib/python3.10/site-packages/soundfile.py", line 658, in __init__
    self._file = self._open(file, mode_int, closefd)
  File "/data/home/acw532/.conda/envs/rms-dev/lib/python3.10/site-packages/soundfile.py", line 1216, in _open
    raise LibsndfileError(err, prefix="Error opening {0!r}: ".format(self.name))
soundfile.LibsndfileError: Error opening <_io.BufferedReader name='/data/scratch/acw532/hardwaxall/79171_B2.mp3'>: Format not recognised.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "apache_beam/runners/common.py", line 1435, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 637, in apache_beam.runners.common.SimpleInvoker.invoke_process
  File "/data/home/acw532/.conda/envs/rms-dev/lib/python3.10/site-packages/klay_beam/transforms.py", line 345, in process
    return (readable_file.metadata.path, np.ndarray([]), sr)
UnboundLocalError: local variable 'sr' referenced before assignment

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/home/acw532/code/kv/klay-beam-jobs/job_rms/bin/run_job_extract_rms.py", line 124, in <module>
    run()
  File "/data/home/acw532/code/kv/klay-beam-jobs/job_rms/bin/run_job_extract_rms.py", line 93, in run
    with beam.Pipeline(argv=pipeline_args, options=pipeline_options) as p:
  File "/data/home/acw532/.conda/envs/rms-dev/lib/python3.10/site-packages/apache_beam/pipeline.py", line 608, in __exit__
    self.result = self.run()
  File "/data/home/acw532/.conda/envs/rms-dev/lib/python3.10/site-packages/apache_beam/pipeline.py", line 585, in run
    return self.runner.run_pipeline(self, self._options)
  File "/data/home/acw532/.conda/envs/rms-dev/lib/python3.10/site-packages/apache_beam/runners/direct/direct_runner.py", line 128, in run_pipeline
    return runner.run_pipeline(pipeline, options)
  File "/data/home/acw532/.conda/envs/rms-dev/lib/python3.10/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 202, in run_pipeline
    self._latest_run_result = self.run_via_runner_api(
  File "/data/home/acw532/.conda/envs/rms-dev/lib/python3.10/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 224, in run_via_runner_api
    return self.run_stages(stage_context, stages)
  File "/data/home/acw532/.conda/envs/rms-dev/lib/python3.10/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 455, in run_stages
    bundle_results = self._execute_bundle(
  File "/data/home/acw532/.conda/envs/rms-dev/lib/python3.10/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 783, in _execute_bundle
    self._run_bundle(
  File "/data/home/acw532/.conda/envs/rms-dev/lib/python3.10/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 1020, in _run_bundle
    result, splits = bundle_manager.process_bundle(
  File "/data/home/acw532/.conda/envs/rms-dev/lib/python3.10/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py", line 1356, in process_bundle
    result_future = self._worker_handler.control_conn.push(process_bundle_req)
  File "/data/home/acw532/.conda/envs/rms-dev/lib/python3.10/site-packages/apache_beam/runners/portability/fn_api_runner/worker_handlers.py", line 379, in push
    response = self.worker.do_instruction(request)
  File "/data/home/acw532/.conda/envs/rms-dev/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py", line 625, in do_instruction
    return getattr(self, request_type)(
  File "/data/home/acw532/.conda/envs/rms-dev/lib/python3.10/site-packages/apache_beam/runners/worker/sdk_worker.py", line 663, in process_bundle
    bundle_processor.process_bundle(instruction_id))
  File "/data/home/acw532/.conda/envs/rms-dev/lib/python3.10/site-packages/apache_beam/runners/worker/bundle_processor.py", line 1056, in process_bundle
    input_op_by_transform_id[element.transform_id].process_encoded(
  File "/data/home/acw532/.conda/envs/rms-dev/lib/python3.10/site-packages/apache_beam/runners/worker/bundle_processor.py", line 237, in process_encoded
    self.output(decoded_value)
  File "apache_beam/runners/worker/operations.py", line 570, in apache_beam.runners.worker.operations.Operation.output
  File "apache_beam/runners/worker/operations.py", line 572, in apache_beam.runners.worker.operations.Operation.output
  File "apache_beam/runners/worker/operations.py", line 263, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
  File "apache_beam/runners/worker/operations.py", line 266, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
  File "apache_beam/runners/worker/operations.py", line 953, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam/runners/worker/operations.py", line 954, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam/runners/common.py", line 1437, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 1526, in apache_beam.runners.common.DoFnRunner._reraise_augmented
  File "apache_beam/runners/common.py", line 1435, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 636, in apache_beam.runners.common.SimpleInvoker.invoke_process
  File "apache_beam/runners/common.py", line 1621, in apache_beam.runners.common._OutputHandler.handle_process_outputs
  File "apache_beam/runners/common.py", line 1734, in apache_beam.runners.common._OutputHandler._write_value_to_tag
  File "apache_beam/runners/worker/operations.py", line 266, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
  File "apache_beam/runners/worker/operations.py", line 953, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam/runners/worker/operations.py", line 954, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam/runners/common.py", line 1437, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 1526, in apache_beam.runners.common.DoFnRunner._reraise_augmented
  File "apache_beam/runners/common.py", line 1435, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 636, in apache_beam.runners.common.SimpleInvoker.invoke_process
  File "apache_beam/runners/common.py", line 1621, in apache_beam.runners.common._OutputHandler.handle_process_outputs
  File "apache_beam/runners/common.py", line 1734, in apache_beam.runners.common._OutputHandler._write_value_to_tag
  File "apache_beam/runners/worker/operations.py", line 266, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
  File "apache_beam/runners/worker/operations.py", line 953, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam/runners/worker/operations.py", line 954, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam/runners/common.py", line 1437, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 1526, in apache_beam.runners.common.DoFnRunner._reraise_augmented
  File "apache_beam/runners/common.py", line 1435, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 636, in apache_beam.runners.common.SimpleInvoker.invoke_process
  File "apache_beam/runners/common.py", line 1621, in apache_beam.runners.common._OutputHandler.handle_process_outputs
  File "apache_beam/runners/common.py", line 1734, in apache_beam.runners.common._OutputHandler._write_value_to_tag
  File "apache_beam/runners/worker/operations.py", line 266, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
  File "apache_beam/runners/worker/operations.py", line 953, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam/runners/worker/operations.py", line 954, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam/runners/common.py", line 1437, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 1526, in apache_beam.runners.common.DoFnRunner._reraise_augmented
  File "apache_beam/runners/common.py", line 1435, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 636, in apache_beam.runners.common.SimpleInvoker.invoke_process
  File "apache_beam/runners/common.py", line 1621, in apache_beam.runners.common._OutputHandler.handle_process_outputs
  File "apache_beam/runners/common.py", line 1734, in apache_beam.runners.common._OutputHandler._write_value_to_tag
  File "apache_beam/runners/worker/operations.py", line 266, in apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
  File "apache_beam/runners/worker/operations.py", line 953, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam/runners/worker/operations.py", line 954, in apache_beam.runners.worker.operations.DoOperation.process
  File "apache_beam/runners/common.py", line 1437, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 1547, in apache_beam.runners.common.DoFnRunner._reraise_augmented
  File "apache_beam/runners/common.py", line 1435, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 637, in apache_beam.runners.common.SimpleInvoker.invoke_process
  File "/data/home/acw532/.conda/envs/rms-dev/lib/python3.10/site-packages/klay_beam/transforms.py", line 345, in process
    return (readable_file.metadata.path, np.ndarray([]), sr)
UnboundLocalError: local variable 'sr' referenced before assignment [while running 'LoadAudio']