Import moar Crops

[roryaronson]

I compiled an insane list of 96k Crop scientific names and associated common names.

Spreadhseet CSV

A few situations to look out for during import:

• Duplicate Crops between what is already in OpenFarm and what is in this set
• Common names in this set associated with scientific names of Crops that are already in OpenFarm

[simonv3]

@rickcarlino I'm not sure what the status is of the XML spreadsheet import. Did you ever get that working? Would it be relatively easy to parse this with that?

[RickCarlino]

Oh cool. The last XML file we had (wish I could remember the name) was actually less useful than its size would imply. A lot of the data was just cruft that didn't apply to OpenFarm. For example, it had base64 encoding images inside of the XML. I will take a look at this after I finish looking at these search bugs on staging.

[simonv3]

Rick did this!

[roryaronson]

I don't think this list was ever imported. Looking at rails admin, we only have ~12k crops in the production database.

[sigu]

@roryaronson , what is the way forward in this?, @simonv3 any thoughts?

[simonv3]

The way I would go about doing this is trying to import the crops from the links Rory put in the header and then writing a rake job to import them to the local db. Once that works (and existing crops don't get overwritten or ignored) we can try running it on staging.

[sigu]

The data we have has two columns, I am not sure which fields they represent. How should we treat them?

[anyangocynthia]

closed by mistake 🙈

[roryaronson]

The first column is the binomial name of the crop. The second column is for comma-separated common names, of which there may be none available, one, or many.

For example, for this crop, the binomial name is Solanum lycopersicum, while the common names include Tomato and Tomate.

[sigu]

According to the validations, name is required for successful saving of the crop. @roryaronson also kindly have a look at my open PR and give feedback

[roryaronson]

@sigu I suppose we could only import crops with at least one common name, and use that value as the OpenFarm Crop Name.

I'm not sure it would be a good idea to import crops and use the binomial name as the OpenFarm Crop Name. Just because that might be confusing. But, it is also probably true that a lot of crops don't have a common name because they aren't very common! In which case, it does make sense to import them and use the binomial name as the OpenFarm Crop Name. Plus, I think it is more important to have more information to start which can slowly be edited, rather than no information and hope it gets added at some point.

So, I guess it makes sense to import everything. If there are any common names, use the first one as the OpenFarm Crop Name. If there are no common names, use the binomial name as the OpenFarm Crop Name. Does that make sense?

In regards to your PR, I'm not a software developer myself so I can't help there 😬

[sigu]

Just to summarise my understanding

• if it has a common name, we use it
• it does not have a common name, use binomial as the common
• I will be assuming the OpenFarm Name is the name db property @simonv3 in this case

Clarification

• should we import both files above or its the same file in different formats?

[roryaronson]

@sigu I think you understand correctly. Let me just clarify:

• If the new crop has at least one common name, use the first one as the OpenFarm Crop Name AND enter all of them into the common names field.
• If the new crop does not have a common name, use the binomial name as the OpenFarm Crop Name AND enter that into the binomial name field.

The "fields" I'm referring to being the ones I see on the crop edit pages, like this one. I'm not exactly sure what those correspond to in the db properties.

Clarification: Both files are the same, just in different formats (excel and CSV)

[sigu]

Then its done now 😄 , and approved