search

mysociety / alaveteli

Provide a Freedom of Information request system for your jurisdiction
https://alaveteli.org
Other
385 stars 195 forks source link

Projects gives users the same extractable request after submission #7631

Closed garethrees closed 1 year ago

garethrees commented 1 year ago

I filled out this extraction:

Screenshot 2023-03-08 at 17 25 49

Submitted, and then got taken back to exactly the same request:

Screenshot 2023-03-08 at 17 26 04

Here's some investigation I did:

Here’s my session data (extracted via https://makandracards.com/makandra/441195-rails-decrypt-a-session-cookie) – a few bits redacted.

decrypt_session 'REDACTEDwK05NNDJXSXMxL1pBeVNBK0VoWGczTEtrK01jUmN3bzc4WURtbWxyUTJCa2E1eUFHTlZtRU16T2JWYktydU5YNjBKMFh0WWtXcEw4QkNNcjh1U1lVSTNXeWNJOEhCQjdpaVBNZVowdnB3YzdXVTNsZEQ0UDVxUTBmakNZU3Y5OThhV1NqczhVUU5NdDRKdlRRN3dzRUVUNkhhdzk4Y3FBUTFaZHN4NFEvVnNoK1BKYlpZMTdpVlVOS1EyVWdVUlhNMHZmVTY4UUY2YmIxVzJIYkViUWlDM1J2anpwbWFRWisrQW54SnR1bEtCTUxDbjJjU2
cyZlZvd2F0UUJNNGUxbFNveVdxSGpRa0NuOW9VMURKc0NUcUVwY3ZKR1k1SklMVHNSSUcwOU9aamJVL3drRnVXYjlxVTlaVlI3emJPQ25hMHBybVY4MlFpTHd3MREDACTEDZmhCejVIS2I1RWJESkY2WUZYRGwvK0IwL2doamFxM2NlNlpMNzF1YzdIUWhFcGpFODRiUXJGOEhzSjhCZjF5ZmwwWE8rZGk4NVdDaFRGV1drU3RvSnd0dCtNbTF0YTFXRjhzTlRTajg5MEUrYWJ0V01sR3pFZDd1c29oakRsd05Xb3VvdlFyRS9HSXRiV2czdldLSDBEV1NMQS94b2JiMXFLMT
JRenhSTm15OWxvbjhjdURyZ0sreVpYcy9BcXdjaVpyVnNrbTdpb1ZwV2dsTjQxUTh6WXJ6YW5mYkZVZEJpUGtreTJOWGVRckZvWnErSHhmWld2ZXJWTnhLTk9iZ0xLdGdvMlNNWnB4N2NMRUU2cEZRL0pBVDFiSEZhM3ZOQ3J6TzMwRlNucFUrWWFkeEZiWDZEMCtzaVVWbFlDb0FFazBKa0xvaEdPMGxMM1QzYW91dlNQdDJFZFdZWFUzT2ZEeWRVT0pyWVRiYUFMSTNxY3Q1TXhSS3RkQjFzVjRVTCt6TWExOG1vdVdxMjN3dTIxamJEcS9TaWhxcUZreDRIdzk0WG1keUZlSVNJRDN0QzMySl
A3VlZNSGhsZ0huODlnVDhUNk44ZDdwWWpPQVpib042YlBWcUQzb1JUM3YrSGM1eVJaR1hVaGZtNFJGS3lkb3JFSUFEbEpPT2NYZVNUVzVOcTZWMTVpUzdYNmM1WGlRNDd2MlVCKzRCd0UxUEtKbldzd3VQYXEvQU1jUXp2ODdSUlFtZkYvbzc4ZmF2SytWbVpuYjJCOENZUmpMeitxWEEvSTFJUS8wNFVhbWtNbVVFVTVxaEJDUC9ZUnkyblhhRkRtRWpGY2pkT0RlK1NMR2ltTXozU1gvcHZWSXRBUjhxcERoUGl5NC9MWWNLbmlGZ3VDMEk3S0pIcHdqWk1nWGlqalpSZDlPQ3U3ejVaYnh4eU
Nla0ZyU0hzUGxWWFcwR1VqaTJUMDRMV0dWYllRYkE3MEUrdkhLVVFFVkU5VXZSWEFVZ245YnlFOWdMZnEwelVXRzRQbi9XbWRZSXJ5R3pyZVAwYy84SUt6bWdKSmlwQ3Q1QlBWcElWc3pQQk5PTFZHK3FxMzA5RXVvVldKVzRkYjk1dlF0Qm91NzFwbDVsQlZYbjd2UGJLckdnTUU0QkdndHZMWlBSZnlZajVHY2RJQmh6eUxDbWl1aGducnU3VGt0SVFVRmZUM0RDcjhXNUFyTDZpcnhGaTh6MG5QY3c3djFiTzZTYTlxUGQ4VDVIQUdUeTRIZVVsb0lqWUlPL2JZRVR3TCtsbTE4UmkvOU1ERk
9OQ3YvVTErSzZlbUNEQW5kNzA1Q0VDdXgyR2dGbTg1NzRHNGRWSlJqdkM0Tkc5Vi8rZUFIdHdYN3FBNGRpM1lBTHp2Zkt4bW40ZEhQU3J3NlF1eTJ0VndGS3VRb1gxRGNsQkUwaFhIM0l1YURYZXlZb01iUDJGOFZ5dW1mc25GVWUycE5LMFBQSWw5S3BXbGQrM0lVRmFoZHNFMkdDZUxoWXVIZC9OKzM1QUNSUWx0WlFFNlQvN2FxdXIyRlRwTGN0N0EwLS0wYnNPRDlodTBORnQ1SEU2ZWJzTDhRPT0%3D--f2804cc2e3034c9d84fa12e3479b48b9bf7d1141'
=>
{"session_id"=>"REDACTED",
 "locale"=>"en",
 "_csrf_token"=>"REDACTED",
 "request_game"=>"2023-03-08T09:43:51.825+00:00",
 "remember_me"=>false,
 "projects"=>
  {"7"=>{"extractable"=>{"current"=>"865950", "skipped"=>[]}},
   "8"=>{"classifiable"=>{"current"=>nil, "skipped"=>[]}, "extractable"=>{"current"=>"865964", "skipped"=>[]}},
   "6"=>{"extractable"=>{"current"=>"865841", "skipped"=>[]}},
   "11"=>{"classifiable"=>{"current"=>"938556", "skipped"=>[]}},
   "12"=>{"classifiable"=>{"current"=>"937676", "skipped"=>[]}, "extractable"=>{"current"=>nil, "skipped"=>[]}},
   "13"=>{"classifiable"=>{"current"=>"937858", "skipped"=>[]}},
   "14"=>{"classifiable"=>{"current"=>"937048", "skipped"=>[]}, "extractable"=>{"current"=>"936954", "skipped"=>[]}},
   "15"=>{"classifiable"=>{"current"=>"937090", "skipped"=>[]}},
   "16"=>{"classifiable"=>{"current"=>"938687", "skipped"=>[]}},
   "22"=>{"extractable"=>{"current"=>"925751", "skipped"=>[]}},
   "17"=>{"extractable"=>{"current"=>"936700", "skipped"=>[]}}},
 "csrf"=>"REDACTED",
 "user_id"=>60565,
 "post_redirect_token"=>"REDACTED",
 "ttl"=>"2023-03-08T17:26:01.816+00:00",
 "using_admin"=>1,
 "admin_name"=>"gareth_rees_3"}

The request that I just submitted an extraction for is 936954

So looking at this specific project

"14"=>{"classifiable"=>{"current"=>"937048", "skipped"=>[]}, "extractable"=>{"current"=>"936954", "skipped"=>[]}},

That request is still being picked up in the extractable call, even though I just submitted an extraction

Project.find(14).info_requests.extractable.pluck(:id).include?(936954)
# => true

which means we’re not falling back to sample where we call find_current in the queue https://github.com/mysociety/alaveteli/blob/develop/app/models/project/queue.rb#L47

I’m user 60565. Last submission for project:

Project.find(14).submissions.last
# => #<Project::Submission:0x00007f3f8b0fd0a8 id: 8120, project_id: 14, user_id: 60565, resource_type: "Dataset::ValueSet", resource_id: 3038, created_at: Wed, 08 Mar 2023 17:26:01.111293000 GMT +00:00, updated_at: Wed, 08 Mar 2023 17:26:01.111293000 GMT +00:00, info_request_id: 936954>

Shows the submission was for the dudley request (936954) https://www.whatdotheyknow.com/admin/requests/936954

Let’s look up that dataset – should the resource type/id be nil? can’t remember

Dataset::ValueSet.find(3038)
# => #<Dataset::ValueSet:0x00007f3f8a8fbdf0 id: 3038, resource_type: nil, resource_id: nil, dataset_key_set_id: 14, created_at: Wed, 08 Mar 2023 17:26:01.081864000 GMT +00:00, updated_at: Wed, 08 Mar 2023 17:26:01.081864000 GMT +00:00>

Keyset is mapped to the correct project:

Dataset::KeySet.find(14)
# => #<Dataset::KeySet:0x00007f3f8ac6a180 id: 14, resource_type: "Project", resource_id: 14, created_at: Wed, 18 Jan 2023 09:50:58.749025000 GMT +00:00, updated_at: Wed, 18 Jan 2023 09:50:58.749025000 GMT +00:00>

Submitted values look good:

Dataset::Value.where(dataset_value_set_id: 3038)
# =>
[#<Dataset::Value:0x00007f3f8acf6db0 id: 18871, dataset_value_set_id: 3038, dataset_key_id: 92, value: "4643", notes: nil, created_at: Wed, 08 Mar 2023 17:26:01.091514000 GMT +00:00, updated_at: Wed, 08 Mar 2023 17:26:01.091514000 GMT +00:00>,
 #<Dataset::Value:0x00007f3f8a4c4610 id: 18872, dataset_value_set_id: 3038, dataset_key_id: 93, value: "3", notes: nil, created_at: Wed, 08 Mar 2023 17:26:01.101457000 GMT +00:00, updated_at: Wed, 08 Mar 2023 17:26:01.101457000 GMT +00:00>,
 #<Dataset::Value:0x00007f3f8b3464b8 id: 18873, dataset_value_set_id: 3038, dataset_key_id: 94, value: "Interim Lead for Climate Change,\r\nWaste Strategy and Disposal Manager,\r\nEnergy Manager", notes: nil, created_at: Wed, 08 Mar 2023 17:26:01.104129000 GMT +00:00, updated_at: Wed, 08 Mar 2023 17:26:01.104129000 GMT +00:00>,
 #<Dataset::Value:0x00007f3f8b346418 id: 18874, dataset_value_set_id: 3038, dataset_key_id: 95, value: "0", notes: nil, created_at: Wed, 08 Mar 2023 17:26:01.106781000 GMT +00:00, updated_at: Wed, 08 Mar 2023 17:26:01.106781000 GMT +00:00>,
 #<Dataset::Value:0x00007f3f8b346378 id: 18875, dataset_value_set_id: 3038, dataset_key_id: 96, value: "", notes: nil, created_at: Wed, 08 Mar 2023 17:26:01.109279000 GMT +00:00, updated_at: Wed, 08 Mar 2023 17:26:01.109279000 GMT +00:00>]

replicated the query in irb

irb(main):030:0> scope = project.info_requests.where(described_state: %w(successful partially_successful)).left_joins(:extraction_project_submissions).classified
=>
[#<InfoRequest:0x00007f3f8aa5c780
...
irb(main):031:0> extractable = scope.where(project_submissions: { id: nil }).or(scope.where.not(project_submissions: { project: project }))
=>
[#<InfoRequest:0x00007f3f8b3abea8
...
irb(main):033:0> extractable.pluck(:id).include?(936954)
=> true

this is where my brain and sql knowledge are faltering. basically, I think there’s a problem with https://github.com/mysociety/alaveteli/commit/2e9c40dd3be177fe562405a51822ea1715a837db

gbp commented 1 year ago

should the resource type/id be nil? can’t remember

This was a bit of a surprise to me but yes it appears this isn't used at all in production. I can't recall why, but it looks like we added Project::Submission#info_request_id which stores the association to InfoRequest to Dataset::ValueSet via Project::Submission#resource.