Closed crisr15 closed 1 year ago
I uploaded the file above and had no problems with any files being uploaded. 🤔 in the screenshot below, the works with 's
added to the end of the title are the ones whose file names ended in XXXs.jpg
the test in my comment above was done by uploading a csv. I then imported the celtics.csv file above using the server path upload method. it also uploaded fine.
I am unable to reproduce this error locally so far.
@rachelihoward can you upload theULUA 2006_003_008_064
"a-s" files that are related to the "owen_compoundObjects_sample.csv" file, into this folder of the google drive?
since I'm unable to reproduce the error locally with my own files, I'd like to see if I can reproduce the error with files you all are using.
also, can you please take a look at my question to you and/or randy in the samvera channel?
by happenstance, I believe I've come across the actual issue. I think the problem is related to when the source_identifier
ends in "s", not the file.
still troubleshooting.
Alisha,
I just requested permissions for that drive.
Randy is working on setting up a user account – what were you planning to use it for?
Rachel
From: Alisha Evans @.> Sent: Wednesday, March 29, 2023 12:23 PM To: scientist-softserv/louisville-hyku @.> Cc: Howard, Rachel @.>; Mention @.> Subject: Re: [scientist-softserv/louisville-hyku] works with a filename that includes the letter “s” before the filetype are skipped (Issue #172)
You don't often get email from @.**@.>. Learn why this is importanthttps://aka.ms/LearnAboutSenderIdentification CAUTION: This email originated from outside of our organization. Do not click links, open attachments, or respond unless you recognize the sender's email address and know the contents are safe.
@rachelihowardhttps://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Frachelihoward&data=05%7C01%7Crachel.howard%40louisville.edu%7Cdc803978e09a4923e16d08db3071d73f%7Cdd246e4a54344e158ae391ad9797b209%7C0%7C0%7C638157037771530527%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=HfZnmYQVT27VRaZpquhKm%2BzQMfz0szqFNTt5gJOCmXg%3D&reserved=0 can you upload theULUA 2006_003_008_064 "a-s" files that are related to the "owen_compoundObjects_sample.csv" file, into this folderhttps://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdrive.google.com%2Fdrive%2Ffolders%2F1q9d-C4Y34EqPB-HbywmCgIWCqFT8qTsr&data=05%7C01%7Crachel.howard%40louisville.edu%7Cdc803978e09a4923e16d08db3071d73f%7Cdd246e4a54344e158ae391ad9797b209%7C0%7C0%7C638157037771530527%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=i%2Fdq1iLJjhF%2FthhsXxvsDapqgtJ4VmmWFmy9zCXh9js%3D&reserved=0 of the google drive?
since I'm unable to reproduce the error locally with my own files, I'd like to see if I can reproduce the error with files you all are using.
also, can you please take a look at my question to you and/or randy in the samvera channelhttps://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsamvera.slack.com%2Farchives%2FC034S4UF53L%2Fp1680042780083409&data=05%7C01%7Crachel.howard%40louisville.edu%7Cdc803978e09a4923e16d08db3071d73f%7Cdd246e4a54344e158ae391ad9797b209%7C0%7C0%7C638157037771530527%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=SHBj8Qx5MWCJM9YFpPG5rfamTXJPcdkcD0zzW6eyv6A%3D&reserved=0?
— Reply to this email directly, view it on GitHubhttps://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fscientist-softserv%2Flouisville-hyku%2Fissues%2F172%23issuecomment-1488918323&data=05%7C01%7Crachel.howard%40louisville.edu%7Cdc803978e09a4923e16d08db3071d73f%7Cdd246e4a54344e158ae391ad9797b209%7C0%7C0%7C638157037771686765%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=OElcCSIsYL1rPycdYx7luDaNPOPjGDIIJnSpsGuVXhE%3D&reserved=0, or unsubscribehttps://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FA5TKJLHAOP2R4XCEPREFMB3W6ROVVANCNFSM6AAAAAAUWUK2IU&data=05%7C01%7Crachel.howard%40louisville.edu%7Cdc803978e09a4923e16d08db3071d73f%7Cdd246e4a54344e158ae391ad9797b209%7C0%7C0%7C638157037771686765%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=6756MRVSRFzYmYYVD2npX%2FpVywN7wBCSazeiI%2BnDml8%3D&reserved=0. You are receiving this because you were mentioned.Message ID: @.**@.>>
That's great news, Alisha!
Sending files to the Google Drive now - it says it will take over an hour. Also, please remove ULUA 2006_003_008_230t from the csv. I don't know why I eliminated the parent and the rest of the children for that one from the sample set but it's not useful to just have that one child record. I think there are plenty of other examples without it!
i = Bulkrax::Importer.find(7)
e = i.entries.last
begin
e.factory.run
rescue
e
end
running the above in my rails console has led me to Hyrax::Actors::TransactionalRequest#create. this method returns false
. according to the comment above the method, returning true means that the create
was successful. so I think this is where the problem lies. not sure why though.
according to the console, next_actor
inside transactional request should be Hyrax::Actors::OptimisticLockValidator. that class doesn't have a create
method in hyrax 2.9.6
though. 🤔
I stepped through each method in the actor stack, seen above, and the part that's failing is the rails #save
method in BaseActor. according to apidock: "If perform_validation is true validations run. If any of them fail the action is cancelled and save returns false." I already knew that the validation was failing, but I assumed it was because the "s" was getting dropped somewhere prior to the save. however, per the below....the "s" is still present on the identifier
property.
so far I'm unable to find where the validation is being set in the bulkrax or lv code.
# /usr/local/bundle/gems/hyrax-2.9.6/app/actors/hyrax/actors/base_actor.rb:68 Hyrax::Actors::BaseActor#save
67: def save(env)
68: env.curation_concern.save
69: end
env.curation_concern.save => false
# note that the identifier is still plural...
[15] pry(#<Hyrax::Actors::TextActor>)> env.curation_concern
=> #<Text id: nil, head: [], tail: [], depositor: "admin@example.com", title: ["Kevon Looney"], date_uploaded: "2023-03-30 15:37:32", date_modified: "2023-03-30 15:37:34", state: nil, proxy_depositor: nil, on_behalf_of: nil, arkivo_checksum: nil, owner: nil, is_child: nil, slug: "kevons", alternative_title: [], collection_information: [], contributor_role: ["publisher"], creator_role: [], date_digital: nil, digitization_specification: "digitization_specification", format: "", location: [], media_type: ["Text"], mesh: [], ordering_information: nil, people_represented: [], resource_query: [], issue: "No. 2.", searchable_text: "Looney always holds us down!", table_of_contents: nil, volume: "Vol 2.", label: nil, relative_path: nil, import_url: nil, resource_type: ["Newspapers"], creator: [], contributor: ["Adam Silver"], description: ["Dub nation"], keyword: [], license: [], rights_statement: [], publisher: ["repository"], date_created: ["2/9/2022"], subject: ["This is the subject"], language: [], identifier: ["Kevons"], based_near: [], related_url: [], bibliographic_citation: [], source: ["The source"], access_control_id: nil, representative_id: nil, thumbnail_id: nil, rendering_ids: [], admin_set_id: "admin_set/default", embargo_id: nil, lease_id: nil>
[16] pry(#<Hyrax::Actors::TextActor>)> env.curation_concern.valid?
=> false
[17] pry(#<Hyrax::Actors::TextActor>)> env.curation_concern.errors.messages
=> {:identifier=>["must be unique"]}
[18] pry(#<Hyrax::Actors::TextActor>)> step
From: /usr/local/bundle/gems/active-fedora-12.2.3/lib/active_fedora/associations/builder/orders.rb:44 ActiveFedora::Associations::Builder::Orders::FixFirstLast#save:
43: def save(*args)
=> 44: super.tap do |result|
45: apply_first_and_last if result
46: end
47: end
[18] pry(#<Text>)> step
From: /usr/local/bundle/gems/active-fedora-12.2.3/lib/active_fedora/validations.rb:50 ActiveFedora::Validations#save:
49: def save(options = {})
=> 50: perform_validations(options) ? super : false
51: end
[18] pry(#<Text>)> perform_validations(options)
=> false
@bkiahstroud suggested I see if this same error happens when creating a work manually; it does. this confirms that the problem is somewhere in the hyrax pathway.
I created a work with a title
and item number
of "single". no file or any other metadata. I then tried creating a second work with a title
and item number
of "singles". it failed with the error below. I then created a third work with a title
and item number
of "doubles". that succeeded. so having an "s" at the end of the identifier alone isn't a problem. it's when that identifier without an "s" is the same as another work that already exists.
this already appeared to be true because identifier
is a property in Hyrax::BasicMetadata. it's created by rdf-vocab, but I don't see any validation rules yet.
I went back through the actor stack and discovered that the identifier gets set somewhere in Hyrax::Actors::BaseActors#create. trying to save the new curation_concern
, fails.
further down the rabbit hole...the problem more closely lies somewhere in the ActiveFedora::Validations#run_validations! method calls.
when I entered ActiveFEDORA::Validations#valid?, it called super
and led me to ActiveMODEL::Validations#valid?. once there, the errors
object had empty hashes for the details
and messages
values. I stepped into the next methods, which was a series of callbacks kicked off by run_validations!
(in ActiveFEDORA), and still didn't see anywhere where an error was set. however, as ActiveMODEL::Validations#valid? wrapped up and brought me back to ActiveFEDORA::Validations#valid?, errors.details
and errors.messages
had data.
further investigation has determined that line 132 of ActiveSupport::Callbacks#run_callbacks(v5.2.7.1) is where the error is being set. I'm unsure though what block was given and where. also, if a block was given, why did it yield false?
I looked at this ticket with jeremy and we found the problem. CustomSlugs::SlugBehavior#check_slug (in the lv codebase) was returning the error. that file is included in each of the model and collection classes. the method in question, is run as a validation for slugs.
we updated line 36 of the above method from possible_duplicates.count.positive?
to possible_duplicates.any? { |pd| pd.slug == slug }
. the result can be seen below.
next I tried to create works manually to see if the problem was resolved. it allowed me to create a work with an identifier and title of beat
followed by a work with an identifier and title of beats
. 🎉 however, while the works were created correctly, the beats
title and identifier do not display correctly.
when looking at the presenter for the "beats" work, several solr properties are saved as "beat" instead.
looking for the solr document in the solr dashboard shows that the title and identifier values show "beats", which is correct. however, when I added an override for #initialize
into Hyrax::WorkShowPresenterDecorator, and refreshed http://hyku.test/concern/texts/beats, the value of solr_document
had the title and identifier values as "beat". 😩
it looks like the wrong solr doc is being passed to the presenter. maybe the same where
method is being called somewhere and it's causing this issue?
def initialize(solr_document, current_ability, request = nil)
binding.pry
@solr_document = solr_document
@current_ability = current_ability
@request = request
end
figured out the problem! ref: #182
reopening because I'm adding specs in a different pr.
@DiemBTran this is ready for staging qa! you can use "/app/samvera/hyrax-webapp/tmp/originals/two_up/two_up.csv" as the server path.
diem is out sick today so I've kicked off importer 2 on staging.
Please tell me it’s not on our hyku-docker-dev site! I have an import still in process there – I started it on March 30 and it’s not yet complete. I was wondering what was taking so long.
From: Alisha Evans @.> Sent: Monday, April 10, 2023 10:54 AM To: scientist-softserv/louisville-hyku @.> Cc: Howard, Rachel @.>; Mention @.> Subject: Re: [scientist-softserv/louisville-hyku] works with a filename that includes the letter “s” before the filetype are skipped (Issue #172)
CAUTION: This email originated from outside of our organization. Do not click links, open attachments, or respond unless you recognize the sender's email address and know the contents are safe.
diem is out sick today so I've kicked off importer 2https://louisville-hyku-staging.notch8.cloud/importers/11?locale=en on staging.
— Reply to this email directly, view it on GitHubhttps://github.com/scientist-softserv/louisville-hyku/issues/172#issuecomment-1501914155, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A5TKJLAYSYCFI4RFP2IN3E3XAQNJPANCNFSM6AAAAAAUWUK2IU. You are receiving this because you were mentioned.Message ID: @.**@.>>
@rachelihoward do not fear, the import is on https://louisville-hyku-staging.notch8.cloud. 🙂
Phew! Thanks for clarifying…
From: Alisha Evans @.> Sent: Monday, April 10, 2023 10:58 AM To: scientist-softserv/louisville-hyku @.> Cc: Howard, Rachel @.>; Mention @.> Subject: Re: [scientist-softserv/louisville-hyku] works with a filename that includes the letter “s” before the filetype are skipped (Issue #172)
CAUTION: This email originated from outside of our organization. Do not click links, open attachments, or respond unless you recognize the sender's email address and know the contents are safe.
@rachelihowardhttps://github.com/rachelihoward do not fear, the import is on https://louisville-hyku-staging.notch8.cloudhttps://louisville-hyku-staging.notch8.cloud/. 🙂
— Reply to this email directly, view it on GitHubhttps://github.com/scientist-softserv/louisville-hyku/issues/172#issuecomment-1501918552, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A5TKJLBPVNCGXCABRP25HY3XAQNXPANCNFSM6AAAAAAUWUK2IU. You are receiving this because you were mentioned.Message ID: @.**@.>>
the importer above failed because the deploy to main with the necessary code changes failed on april 6.
after updating the volumes on staging and redeploying, I was able to successfully rerun the importer using the csv and files in the zip above. 🎉
@rkuehn-uofl @rachelihoward please take a look at https://github.com/rkuehn-uofl/hyku/pull/25 and qa!
@alishaevn we're testing the latest code. CC: @rachelihoward
@alishaevn we successfully tested this on our dev server (which was pretty much empty)! The soon-to-be-production server, with 81K plus items on it, has become so buggy that we can't successfully test anything on it.
Summary
During bulkrax ingest some children have alphanumeric filenames, and I’ve noticed that when a filename includes the letter “s” before the filetype it is skipped – is there something in the code that treats an “s” before a filetype as a stop word? See, for example, https://hyku-docker-dev.library.louisville.edu/concern/images/ulua_2006_003_008_299/ - it skips from ulua_2006_003_008_299r to ulua_2006_003_008_299t
Acceptance Criteria
Testing Instructions
Notes
When looking at the file mentioned above I noticed that the file ulua_2006_003_008_299f was also missing.
Confirmed in meeting that 's' is the repeated problem. Do not worry about f missing currently on this one item.