Support `Import from CSV` option

[apexdodge]

79 is a pre-req

Add a new item in this dropdown box: Import from CSV

The header column in the CSV should match the DeveloperName of the field it should map to.

There should be two options:

• Insert new records only
• Insert and update existing records

[smd-11]

Hey @apexdodge, I would like to work on this.

[apexdodge]

@smd-11 Thank you very much.

Here is a draft of the spec. Let me know your feedback.

1) Add a link below Export to CSV named Import from CSV

2) Only show link if user as Edit permission on the content type.

3) This takes the user to a UI with a radio field of the following options and must select one option.

• Add new records only
• Upsert all records
• Update existing records only

4) File upload field accepts .csv file.

5) User posts the form.

6) Application layer takes in the data, runs validation, and inserts appropriately, updating a status on progress (runs in background).

7) User is redirected to a status screen very similar to that of Export to CSV to see how well the import is progressing.

Once we agree on spec, we can talk about more how best to implement this from a technical perspective.

[smd-11]

I am good

[apexdodge]

@smd-11 Okay great. We can split the task up into 2 parts. We can start with part 1 which I think is the harder of the two actually, the backend.

1) Create a new Command called BeginImportContentItemsFromCsv https://github.com/RaythaHQ/raytha/tree/main/src/Raytha.Application/ContentItems/Commands. Can use the BeginExportContentItemsToCsv command as a starting place.

2) The Command should probably take in the following attributes:

• ShortGuid ViewId
• string ImportMethod ("insert_new_only", "upsert_all_records", "update_existing_only")
• bool ImportAsDraft
• byte[] CsvAsBytes

3) The Handler can actually look a lot like ExportToCsv's handler.

4) BackgroundTask for this should be a lot simpler ExportToCsv's. Most of the logic there can be written from scratch but follow this pattern:

Challenges in this step

• Converting the CsvAsBytes to an ienumerable<ienumerable> so that the data is iterable.
• Get the first row which should be columns of id and the rest of the columns should match a Developer Name as defined by the user in a content type.
• Loop through iterable
  • depending on ImportMethod:
  • if value exists in the id column, get the item, update item
  • if value does not exist in id column, insert item
  • if ImportAsDraft, set entity.IsDraft = true and entity.IsPublished = false. Else, entity.IsDraft = false and entity.IsPublished = true

Some advice, up to you:

• Review docs on background tasks in Raytha: https://docs.raytha.com/articles/background_tasks.html
• Look at CreateContentItem and EditContentItem commands: https://github.com/RaythaHQ/raytha/blob/main/src/Raytha.Application/ContentItems/Commands/CreateContentItem.cs https://github.com/RaythaHQ/raytha/blob/main/src/Raytha.Application/ContentItems/Commands/EditContentItem.cs
• In step 4, at first, do not try to convert bytes to CSV and do any loop through. Just hardcode inserting values, just to see if you can get items to insert at all
• Feel free to create a temporary endpoint on a controller to invoke the command without any UI.

Thank you again for your interest and let me know if you have any questions or concerns.

[smd-11]

Hey @apexdodge ,getting exception at line 49 in QueueHostedService.cs :

May i know what the above file does.

The requested operation requires an element of type 'Number', but the target element has type 'String'.

Can you help with this?

[apexdodge]

@smd-11 This code is responsible for seeing if there were any requests to process a background task, and if there is, it will begin that execution.

I suspect there might be an issue with the BackgroundTask class you may have created as shown here: https://docs.raytha.com/articles/background_tasks.html#create-a-class-that-implements-the-iexecutebackgroundtask-interface -- perhaps a mismatch in pulling the arguments out.

Can you share your BeginImportContentItemsFromCsv file to this point? I think it might help uncover the issue.

[smd-11]

resolved issue.... was using byte instead of byte[]

[smd-11]

@apexdodge are these fields enough to be there in csv file.

Id,Template,title,content,contenttype ,Content item detail view,Test Data,This is test data,pages

[apexdodge]

@smd-11

Good questions 👍🏻 .

contenttype column is not needed because when you click Import from CSV, we are doing it from a View that is already scoped to the content type (hence the ViewId, which we can use to get the contenttype):

The rest are good.

One change on this:

I think this should be the Template's developer name, not label. You can find the developer name of a template like so:

Then you can use something like below (doing it from memory)

var template = _db.WebTemplates.FirstOrDefault(p => p.DeveloperName == templateDeveloperName)

Hope that helps! Let me know if you have any other questions.

[smd-11]

Hey @apexdodge, pushed code to https://github.com/smd-11/raytha/tree/excfix.

[smd-11]

Will create PR if everything is fine

[apexdodge]

@smd-11

Excellent work. There is one place that needs fixing.

We cannot hard code these since users define their own content types and create their own fields. So we are expecting the columns in the CSV to match developer names as shown here:

So you can create your own fields first in the user interface, specify the developer name, and then use CSV import with column names that match those developer names so that you can bulk import data.

Let me know if that makes sense.

[smd-11]

yeah will update code accordingly

[smd-11]

Hi @apexdodge , updated code at https://github.com/smd-11/raytha/tree/excfix .Have a look and let me know if any changes required

[smd-11]

@apexdodge

A few queries:

• should data for attachment field type be uploaded from csv? If yes, what the user will enter in csv "Local path of file"?
• can we have only text data in csv for wysiwyg and long_text field type or images too? if image too then how can it will be entered in csv?
• Also provide me some context for one_to_one_relationship field type

[apexdodge]

@smd-11, yes good questions.

• For v1 of this feature we will not support file upload. I have ideas on how it will work, but it's too much for right now.
• Only text data is fine, or more specifically, the expectation is that the person has html in these fields. So don't worry about images here.
• For one to one relationship, it is expecting the Id of the related object item. Similar to how we have the Id field for Updates, this column should point to the Id of another content item.

Let me know if anything isn't clear from that, I can happily explain.

[smd-11]

@apexdodge created PR for this.

[apexdodge]

@smd-11 Excellent, thank you. I will review this weekend.

[apexdodge]

Branch release-v1.1.2 has been published and I did a little bit of code clean up and testing. It works quite well.

Tomorrow I will merge into main and write up the documentation.

[apexdodge]

Released with v1.1.2!