File sets (intermittently) not attaching to works in Arch production

[kdid]

Description

File sets (intermittently) not attaching to works on Arch production

Here is an example work where files won't attach: https://arch.library.northwestern.edu/concern/generic_works/br86b3901

Done Looks Like

• [ ] problem identified and resolved.

cc: @chrisdaaz

[chrisdaaz]

I don't know if this matters, but I was using the drag-and-drop function for all of works where the files weren't attaching. Arch is supposed to require files for works to be saved, and I get a success message when the file saves, but then I see the yellow alert about the work not having files.

[kdid]

ok. thanks for the info.

[kdid]

@chrisdaaz - We looked into this for several hours yesterday and weren't able to make any progress. We haven't been able to reproduce the error. (we tried dragging and dropping and it worked). There are no errors being logged or thrown that we can see.

Can you keep an eye on whether it keeps happening? If it does, let us know and we'll have to come up with a plan for how we can diagnose what's happening.

[chrisdaaz]

@kdid this problem is still happening with me today. i've asked someone else to see if they have any issues. i'll let you know if they report any issues depositing a work.

would it be helpful if i record a screencast?

[kdid]

Yeah, if you can record a screencast that would be great. Maybe it will show us something.

[chrisdaaz]

It's starting to feel like it's just me, but here's a screencast (windows 10/ brave browser): https://northwestern.box.com/s/4ikxe4m445n2ndqvto4ncnvkb5j4rcv3

Here's the work I attempted: https://arch.library.northwestern.edu/concern/generic_works/zc77sq43h?locale=en

I'll try again on Chrome.

[chrisdaaz]

Here it is (same process) working on Chrome:

[kdid]

So, does this fail 100% of the time when you are using Windows/Brave? And succeed 100% of the time when you are using Chrome?

[chrisdaaz]

not 100% in either browser. also failed for me on windows/edge on a new laptop. could it be a home network problem?

[kdid]

It would seem strange if it were your home network - we're all working on home networks right now I think. But I guess we can't rule it out completely.

[kdid]

Previous issue: https://github.com/nulib/arch/issues/521

Investigation - works with no file sets currently in prod:

{
  "responseHeader": {
    "zkConnected": true,
    "status": 0,
    "QTime": 1,
    "params": {
      "q": "-file_set_ids_ssim:[\"\" TO *]",
      "fl": "id,creator_tesim,system_modified_dtsi",
      "fq": "has_model_ssim:GenericWork",
      "rows": "20"
    }
  },
  "response": {
    "numFound": 17,
    "start": 0,
    "docs": [
      {
        "system_modified_dtsi": "2019-01-29T16:56:40Z",
        "id": "4j03cz701",
        "creator_tesim": [
          "Gowda, Karna"
        ]
      },
      {
        "system_modified_dtsi": "2019-02-26T17:25:23Z",
        "id": "sf268511n",
        "creator_tesim": [
          "Sze-Dai Pang",
          "Zdenek P. Bazant"
        ]
      },
      {
        "system_modified_dtsi": "2019-01-29T17:03:22Z",
        "id": "m613mx627",
        "creator_tesim": [
          "Gowda, Karna"
        ]
      },
      {
        "system_modified_dtsi": "2019-01-29T17:04:57Z",
        "id": "qb98mf57c",
        "creator_tesim": [
          "Gowda, Karna"
        ]
      },
      {
        "system_modified_dtsi": "2019-01-29T17:04:08Z",
        "id": "nv9352905",
        "creator_tesim": [
          "Gowda, Karna"
        ]
      },
      {
        "system_modified_dtsi": "2019-02-26T17:02:25Z",
        "id": "c534fp00z",
        "creator_tesim": [
          "Sze-Dai Pang",
          "Zdenek P. Bazant "
        ]
      },
      {
        "system_modified_dtsi": "2019-01-29T17:30:26Z",
        "id": "2n49t190j",
        "creator_tesim": [
          "Victoria Anne Church"
        ]
      },
      {
        "system_modified_dtsi": "2019-02-26T16:56:21Z",
        "id": "7w62f826z",
        "creator_tesim": [
          "Sze-Dai Pang",
          "Zdenek P. Bazant"
        ]
      },
      {
        "system_modified_dtsi": "2019-01-29T17:30:24Z",
        "id": "ff365544f",
        "creator_tesim": [
          "Victoria Anne Church"
        ]
      },
      {
        "system_modified_dtsi": "2019-02-26T17:09:27Z",
        "id": "g732d9021",
        "creator_tesim": [
          "Sze-Dai Pang",
          "Zdenek P. Bazant "
        ]
      },
      {
        "system_modified_dtsi": "2019-01-29T16:59:38Z",
        "id": "br86b364d",
        "creator_tesim": [
          "Gowda, Karna"
        ]
      },
      {
        "system_modified_dtsi": "2019-01-29T17:06:36Z",
        "id": "td96k255b",
        "creator_tesim": [
          "Gowda, Karna"
        ]
      },
      {
        "system_modified_dtsi": "2021-05-11T20:26:31Z",
        "id": "jd472w89w",
        "creator_tesim": [
          "Caubble, Jared"
        ]
      },
      {
        "system_modified_dtsi": "2021-05-12T02:28:52Z",
        "id": "st74cq95h",
        "creator_tesim": [
          "Huang, Xiaoyu"
        ]
      },
      {
        "system_modified_dtsi": "2021-06-25T00:13:42Z",
        "id": "zc77sq43h",
        "creator_tesim": [
          "Jaglinski, Angelina "
        ]
      },
      {
        "system_modified_dtsi": "2021-06-25T01:55:48Z",
        "id": "0k225b40j",
        "creator_tesim": [
          "Jaglinski, Angelina"
        ]
      },
      {
        "system_modified_dtsi": "2021-06-25T17:03:36Z",
        "id": "br86b3901",
        "creator_tesim": [
          "Kikuchi, Thomas "
        ]
      }
    ]
  },
  "facet_counts": {
    "facet_queries": {},
    "facet_fields": {},
    "facet_ranges": {},
    "facet_intervals": {},
    "facet_heatmaps": {}
  }
}

[chrisdaaz]

i'm still struggling to create works successfully.

from today: https://arch.library.northwestern.edu/concern/generic_works/70795806c?locale=en

from yesterday: https://arch.library.northwestern.edu/concern/generic_works/rx913q23v?locale=en

[bmquinn]

Thanks for the heads up @chrisdaaz, I just checked the Arch queue in Amazon SQS and there aren't any stuck messages (clearing the queue has helped get things moving in the past, but apparently it's not the culprit today). At this point I think the dev team is going to have to group up next week to come up with a plan. We're looking at a few things but swapping out Active Elastic Job with Shoryuken for background job processing is the big one, as well as adding instrumentation/logging to the Hyrax actor stack that performs all of the FileSet attaching in the background so we can get a better sense of what's happening.

[chrisdaaz]

i now think it's a metadata problem. i noticed this afternoon that was i was able to reliably create new works with properly attached filesets by entering a minimal amount of metadata (e.g. test for title, creator, keyword, etc.). after doing that, i can do back and edit the work with the correct metadata. obviously, this is a tedious process for me, but it works for now and i'm able to continue depositing stuff.

the problem seems to only happen consistently when i deposit a new work with metadata copied from VS Code and pasted into the new work form UI. i think there might be something going on with character encoding between VS Code and pasting in the Arch UI. copying metadata from VS Code is the only thing that consistently creates the problem of filesets missing from a newly deposited work.

We're looking at a few things but swapping out Active Elastic Job with Shoryuken for background job processing is the big one, as well as adding instrumentation/logging to the Hyrax actor stack that performs all of the FileSet attaching in the background so we can get a better sense of what's happening.

that sounds like a lot of work. while this problem is annoying for me, it might be due to my idiosyncratic workflows for getting new content into Arch. since the problem is not reproducible, i think it's worth prioritizing the new dataset feature work over this, but i'll defer to you and @davidschober. i suppose it's possible these errors are indicative of more serious bugs waiting to be discovered and the plan you develop may help with those too.

[bmquinn]

That's super helpful! I'm sure all us devs are testing upload with minimal metadata like "test" in all required fields. It's possible you've discovered a character encoding bug with form inputs, and Is totally worth looking into.

[bmquinn]

@chrisdaaz I tried a bunch of different combinations of character encodings and inputs and couldn't reproduce the issue, so I think it'd help if you could send a sample file from VSCode or a screenshot of VSCode showing the text and bottom status bar that shows file encoding and end of line sequence settings, e.g.:

(side note: the little gremlin icons on the left side of the text are from the Gremlins extension which reveals possibly harmful characters such as zero-width spaces and left and right double quotation marks - https://marketplace.visualstudio.com/items?itemName=nhoizey.gremlins)

[chrisdaaz]

screenshot:

file: https://northwestern.box.com/s/g16cc9cc70ad90u8xhd0dkshlva1gwwo

[bmquinn]

Thanks a ton @chrisdaaz I'll run some tests with the file and let you know when I get some results. 💯

[bmquinn]

@chrisdaaz we figured it out! The bug is caused by double spaces in the form fields. I'm checking to see whether it's an upstream bug in Hyrax or not, and looking into solving it on our end as well. Thanks for your patience in letting us get to the bottom of this!

[kdid]

@chrisdaaz - do we need to take any action on the works listed in the comment above ^ that don't have file sets attached?

[chrisdaaz]

@kdid no. thank you for pointing those out to me. i'll figure out how to handle those cases.