search

SETI / rms-opus

PDS OPUS - Outer Planets Data Search Tool
Apache License 2.0
9 stars 7 forks source link

log_analyzer crash #1352

Closed rfrenchseti closed 2 months ago

rfrenchseti commented 3 months ago

The OPUS log analyzer is crashing when creating the session files for 12/2023. All logs and manifests are in Dropbox.

Found 62 log files
Found 196 manifests
Reading /Volumes/opus/shared/apache_logs/tools2_access_log-2023-12-01
Reading /Volumes/opus/shared/apache_logs/tools2_access_log-2023-12-02
Reading /Volumes/opus/shared/apache_logs/tools2_access_log-2023-12-03
Reading /Volumes/opus/shared/apache_logs/tools2_access_log-2023-12-04
Reading /Volumes/opus/shared/apache_logs/tools2_access_log-2023-12-05
Reading /Volumes/opus/shared/apache_logs/tools2_access_log-2023-12-06
Reading /Volumes/opus/shared/apache_logs/tools2_access_log-2023-12-07
Reading /Volumes/opus/shared/apache_logs/tools2_access_log-2023-12-08
Reading /Volumes/opus/shared/apache_logs/tools2_access_log-2023-12-09
Reading /Volumes/opus/shared/apache_logs/tools2_access_log-2023-12-10
Reading /Volumes/opus/shared/apache_logs/tools2_access_log-2023-12-11
Reading /Volumes/opus/shared/apache_logs/tools2_access_log-2023-12-12
Reading /Volumes/opus/shared/apache_logs/tools2_access_log-2023-12-13
Reading /Volumes/opus/shared/apache_logs/tools2_access_log-2023-12-14
Reading /Volumes/opus/shared/apache_logs/tools2_access_log-2023-12-15
Reading /Volumes/opus/shared/apache_logs/tools2_access_log-2023-12-16
Reading /Volumes/opus/shared/apache_logs/tools2_access_log-2023-12-17
Reading /Volumes/opus/shared/apache_logs/tools2_access_log-2023-12-18
Reading /Volumes/opus/shared/apache_logs/tools2_access_log-2023-12-19
Reading /Volumes/opus/shared/apache_logs/tools2_access_log-2023-12-20
Reading /Volumes/opus/shared/apache_logs/tools2_access_log-2023-12-21
Reading /Volumes/opus/shared/apache_logs/tools2_access_log-2023-12-22
Reading /Volumes/opus/shared/apache_logs/tools2_access_log-2023-12-23
Reading /Volumes/opus/shared/apache_logs/tools2_access_log-2023-12-24
Reading /Volumes/opus/shared/apache_logs/tools2_access_log-2023-12-25
Reading /Volumes/opus/shared/apache_logs/tools2_access_log-2023-12-26
Reading /Volumes/opus/shared/apache_logs/tools2_access_log-2023-12-27
Reading /Volumes/opus/shared/apache_logs/tools2_access_log-2023-12-28
Reading /Volumes/opus/shared/apache_logs/tools2_access_log-2023-12-29
Reading /Volumes/opus/shared/apache_logs/tools2_access_log-2023-12-30
Reading /Volumes/opus/shared/apache_logs/tools2_access_log-2023-12-31
Reading /Volumes/opus/shared/apache_logs/tools_access_log-2023-12-01
Reading /Volumes/opus/shared/apache_logs/tools_access_log-2023-12-02
Reading /Volumes/opus/shared/apache_logs/tools_access_log-2023-12-03
Reading /Volumes/opus/shared/apache_logs/tools_access_log-2023-12-04
Reading /Volumes/opus/shared/apache_logs/tools_access_log-2023-12-05
Reading /Volumes/opus/shared/apache_logs/tools_access_log-2023-12-06
Reading /Volumes/opus/shared/apache_logs/tools_access_log-2023-12-07
Reading /Volumes/opus/shared/apache_logs/tools_access_log-2023-12-08
Reading /Volumes/opus/shared/apache_logs/tools_access_log-2023-12-09
Reading /Volumes/opus/shared/apache_logs/tools_access_log-2023-12-10
Reading /Volumes/opus/shared/apache_logs/tools_access_log-2023-12-11
Reading /Volumes/opus/shared/apache_logs/tools_access_log-2023-12-12
Reading /Volumes/opus/shared/apache_logs/tools_access_log-2023-12-13
Reading /Volumes/opus/shared/apache_logs/tools_access_log-2023-12-14
Reading /Volumes/opus/shared/apache_logs/tools_access_log-2023-12-15
Reading /Volumes/opus/shared/apache_logs/tools_access_log-2023-12-16
Reading /Volumes/opus/shared/apache_logs/tools_access_log-2023-12-17
Reading /Volumes/opus/shared/apache_logs/tools_access_log-2023-12-18
Reading /Volumes/opus/shared/apache_logs/tools_access_log-2023-12-19
Reading /Volumes/opus/shared/apache_logs/tools_access_log-2023-12-20
Reading /Volumes/opus/shared/apache_logs/tools_access_log-2023-12-21
Reading /Volumes/opus/shared/apache_logs/tools_access_log-2023-12-22
Reading /Volumes/opus/shared/apache_logs/tools_access_log-2023-12-23
Reading /Volumes/opus/shared/apache_logs/tools_access_log-2023-12-24
Reading /Volumes/opus/shared/apache_logs/tools_access_log-2023-12-25
Reading /Volumes/opus/shared/apache_logs/tools_access_log-2023-12-26
Reading /Volumes/opus/shared/apache_logs/tools_access_log-2023-12-27
Reading /Volumes/opus/shared/apache_logs/tools_access_log-2023-12-28
Reading /Volumes/opus/shared/apache_logs/tools_access_log-2023-12-29
Reading /Volumes/opus/shared/apache_logs/tools_access_log-2023-12-30
Reading /Volumes/opus/shared/apache_logs/tools_access_log-2023-12-31
Parsing input
Writing file /tmp/log_analyzer_results_temp_refresh/logs/2023/OPUS-log-analysis-2023-12.html
Writing sessions to /tmp/log_analyzer_results_temp_refresh/logs/2023/../../.sessions/2023/2023-12/
Traceback (most recent call last):
  File "/opus/src/pds-opus/log_analyzer/log_analyzer.py", line 159, in <module>
    main()
  File "/opus/src/pds-opus/log_analyzer/log_analyzer.py", line 127, in main
    log_parser.run_batch(log_entries_list)
  File "/opus/src/pds-opus/log_analyzer/log_parser.py", line 135, in run_batch
    self.__generate_batch_html_output(host_infos)
  File "/opus/src/pds-opus/log_analyzer/log_parser.py", line 268, in __generate_batch_html_output
    batch_html_generator.generate_output(self._output)
  File "/opus/src/pds-opus/log_analyzer/opus/html_generator.py", line 88, in generate_output
    for line in get_lines():
  File "/opus/src/pds-opus/log_analyzer/opus/html_generator.py", line 71, in get_lines
    for chunk in itertools.chain(output_generator, ['\n']):
  File "/opus/opus_venv/lib/python3.11/site-packages/jinja2/environment.py", line 1354, in generate
    yield self.environment.handle_exception()
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opus/opus_venv/lib/python3.11/site-packages/jinja2/environment.py", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "templates/log_analysis.html", line 273, in top-level template code
    {% include 'histogram.html' %}
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "templates/histogram.html", line 1, in top-level template code
    {% with session_statistics = context.get_session_statistics(),
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opus/src/pds-opus/log_analyzer/opus/html_generator.py", line 278, in get_manifest_download_statistics
    result = ManifestStatus.get_statistics(self._configuration.manifests)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opus/src/pds-opus/log_analyzer/manifest.py", line 155, in get_statistics
    return status.__get_statistics()
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opus/src/pds-opus/log_analyzer/manifest.py", line 131, in __get_statistics
    result2, total2 = self.__get_one_table(lambda entry: (entry.volume_set,))
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opus/src/pds-opus/log_analyzer/manifest.py", line 104, in __get_one_table
    all_items.sort(key=lambda item: grouper(item[1]))
  File "/opus/src/pds-opus/log_analyzer/manifest.py", line 104, in <lambda>
    all_items.sort(key=lambda item: grouper(item[1]))
                                    ^^^^^^^^^^^^^^^^
  File "/opus/src/pds-opus/log_analyzer/manifest.py", line 131, in <lambda>
    result2, total2 = self.__get_one_table(lambda entry: (entry.volume_set,))
                                                          ^^^^^^^^^^^^^^^^
  File "/opus/src/pds-opus/log_analyzer/manifest.py", line 38, in volume_set
    assert match
AssertionError
fyellin commented 3 months ago

Can you remind me of the command you use to run this?

On Mon, Apr 1, 2024 at 9:42 AM Robert French @.***> wrote:

Assigned #1352 https://github.com/SETI/rms-opus/issues/1352 to @fyellin https://github.com/fyellin.

— Reply to this email directly, view it on GitHub https://github.com/SETI/rms-opus/issues/1352#event-12312986061, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABC2LIJ5FGRBSFRSW7CJ3YTY3GE67AVCNFSM6AAAAABFR3TWOOVHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJSGMYTEOJYGYYDMMI . You are receiving this because you were assigned.Message ID: @.***>

rfrenchseti commented 3 months ago 
python log_analyzer.py --cronjob-date 2023-12 --html --dns `cat /opus/src/pds-webserver/opus_servers/cron_files/log_analyzer_ignore_ip.txt` --sessions-relative-directory "../../.sessions/%Y/%Y-%m" -o "/tmp/log_analyzer_results_temp_refresh/logs/%Y/OPUS-log-analysis-%Y-%m.html" --manifest "/Volumes/opus/shared/manifests/*%Y-%m*" "/Volumes/opus/shared/apache_logs/tools_access_log-%Y-%m-%d" "/Volumes/opus/shared/apache_logs/tools2_access_log-%Y-%m-%d"
fyellin commented 2 months ago

The bug is that we assumed that the volume name for a manifest entry was alphanumeric. Starting in December, it seems that the volume names could contain periods and hyphens. The regexp used turned out to be bogus for other reasons, too.

The simplest solution is to replace the regexp with PosixPath, which knows how to parse Unix path names correctly.

rfrenchseti commented 2 months ago

Nothing changed in the past year. What volume names are you seeing that are not alphanumeric?

fyellin commented 2 months ago

/volumes/HSTUx_xxxx_v1.0/HSTU0_9344/DATA/VISIT_06/U6LG0609M.ASC

rfrenchseti commented 2 months ago

Ah. OK that's always been true. I guess no one ever tried to download something from a previous version of a volume before.