[production] Changes in UI not written into backend

[raddirad]

Issue/Feature description

Since a filesystem crash our local instance always gives error 500 whenever I want to change something within the repository setting in the UI.

Here's the output when I want to disable or enable a repository for a package:

I, [2017-08-30T08:02:13.954673 #18610]  INFO -- : [845e3534-ce09-4164-a475-e307fb6da491] [18610:1549012.54] Started PUT "/flag/home:uibmz:opsi:opsi40-experimental/opsi-nagios-plugins?flag=2460" for 192.168.2.16 at 2017-08-30 08:02:13 +0000
I, [2017-08-30T08:02:13.963390 #18610]  INFO -- : [845e3534-ce09-4164-a475-e307fb6da491] [18610:1549012.55] Processing by Webui::RepositoriesController#toggle_flag as HTML
I, [2017-08-30T08:02:13.963569 #18610]  INFO -- : [845e3534-ce09-4164-a475-e307fb6da491] [18610:1549012.55]   Parameters: {"authenticity_token"=>"65NFVAZWAzFLpN3sFJNUEp76Py4fXUNkD3+xlDWH8PMmxeOKa8PVeDHe9eelCR5lIVS2sAb9I9WMa44oJogx9w==", "flag"=>"2460", "project"=>"home:uibmz:opsi:opsi40-experimental", "package"=>"opsi-nagios-plugins"}
I, [2017-08-30T08:02:14.057073 #18610]  INFO -- : [845e3534-ce09-4164-a475-e307fb6da491] [18610:1549012.64]   Rendered models/_package.xml.builder (24.1ms)
I, [2017-08-30T08:02:14.098253 #18610]  INFO -- : [845e3534-ce09-4164-a475-e307fb6da491] [18610:1549012.68] Completed 500 Internal Server Error in 134ms (ActiveRecord: 0.0ms | Backend: 15.1ms | XML: 0.0ms)
F, [2017-08-30T08:02:14.108938 #18610] FATAL -- : [845e3534-ce09-4164-a475-e307fb6da491] [18610:1549012.69]   
F, [2017-08-30T08:02:14.109043 #18610] FATAL -- : [845e3534-ce09-4164-a475-e307fb6da491] [18610:1549012.69] ActiveXML::Transport::Error (<status code="400">
  <summary>/srv/obs/events/lastnotifications: bad last line at /usr/lib/obs/server/BSFileDB.pm line 269.</summary>
</status>
):
F, [2017-08-30T08:02:14.109124 #18610] FATAL -- : [845e3534-ce09-4164-a475-e307fb6da491] [18610:1549012.69]   
F, [2017-08-30T08:02:14.109239 #18610] FATAL -- : [845e3534-ce09-4164-a475-e307fb6da491] [18610:1549012.69] lib/opensuse/backend.rb:179:in `handle_response'
[845e3534-ce09-4164-a475-e307fb6da491] lib/opensuse/backend.rb:92:in `put_or_post'
[845e3534-ce09-4164-a475-e307fb6da491] lib/opensuse/backend.rb:96:in `put'
[845e3534-ce09-4164-a475-e307fb6da491] app/models/package.rb:743:in `write_to_backend'
[845e3534-ce09-4164-a475-e307fb6da491] app/models/package.rb:724:in `store'
[845e3534-ce09-4164-a475-e307fb6da491] app/controllers/webui/repositories_controller.rb:207:in `block in toggle_flag'
[845e3534-ce09-4164-a475-e307fb6da491] app/controllers/webui/repositories_controller.rb:204:in `toggle_flag'
[845e3534-ce09-4164-a475-e307fb6da491] config/initializers/wrap_parameters.rb:38:in `call'
I, [2017-08-30T08:02:25.498836 #1669]  INFO -- : [1669:1642814.64] 2017-08-30T08:02:25+0000: [Worker(delayed_job.1030 host:obs pid:1669)] Job Event::NotifyBackends#send_not_in_queue (id=113764) RUNNING
I, [2017-08-30T08:02:25.504944 #1669]  INFO -- : [1669:1642814.65] 2017-08-30T08:02:25+0000: [Worker(delayed_job.1030 host:obs pid:1669)] Job Event::NotifyBackends#send_not_in_queue (id=113764) COMPLETED after 0.0058
I, [2017-08-30T08:02:25.511008 #1669]  INFO -- : [1669:1642814.66] 2017-08-30T08:02:25+0000: [Worker(delayed_job.1030 host:obs pid:1669)] Job SendEventEmails#perform (id=113765) RUNNING
I, [2017-08-30T08:02:25.521926 #1669]  INFO -- : [1669:1642814.67] 2017-08-30T08:02:25+0000: [Worker(delayed_job.1030 host:obs pid:1669)] Job SendEventEmails#perform (id=113765) COMPLETED after 0.0107
I, [2017-08-30T08:02:25.523591 #1669]  INFO -- : [1669:1642814.67] 2017-08-30T08:02:25+0000: [Worker(delayed_job.1030 host:obs pid:1669)] 2 jobs processed at 62.1158 j/s, 0 failed

rpm -qa | grep obs
obs-api-2.8.2-1.2.noarch
obs-service-download_url-0.1-6.1.noarch
obs-server-2.8.2-1.2.noarch
obs-service-recompress-0.3.1+git20160217.7897d3f-8.1.noarch
obs-service-obs_scm-common-0.7.0.1492101301.747de50-6.3.1.noarch
obs-service-set_version-0.5.6-5.3.1.noarch
obs-service-obs_scm-0.7.0.1492101301.747de50-6.3.1.noarch
obs-service-extract_file-0.3-6.1.noarch
obs-worker-2.8.2-1.2.noarch
obs-source_service-2.8.2-1.2.noarch
obs-service-verify_file-0.1.1-21.1.noarch
obs-common-2.8.2-1.2.noarch
obs-service-download_src_package-0.2-3.1.noarch
obs-service-snapcraft-0.7.0.1492101301.747de50-6.3.1.noarch
obs-service-format_spec_file-20170204-2.3.1.noarch
obs-service-tar_scm-0.7.0.1492101301.747de50-6.3.1.noarch
obs-service-source_validator-0.6+git20170111.82ea590-13.3.1.noarch
obs-service-tar-0.7.0.1492101301.747de50-6.3.1.noarch
obs-productconverter-2.8.2-1.2.noarch
obs-signd-2.3.0-2.1.x86_64
obs-service-download_files-0.5.1.git.1455712026.9c0a4a0-7.1.noarch
obs-utils-2.8.2-1.2.noarch

[mschnitzer]

According to the log, it seems to be an issue in BSFileDB.pm @M0ses @lethliel @mlschroe

[lethliel]

@raddirad

This looks like your local files are somehow corrupted. Can you check the lastevents file in /srv/obs/events and check if the file looks sane?

[adrianschroeter]

please, github issues are no for doing support. Please go to the mailing list or ask for commercial support for this.

[drizt]

@lethliel just want to say thanks. I got Server returned an error: HTTP Error 400: Bad Request with my OBS instance. After I removed all in /srv/obs/events all works again.