Can a batch upload be created for multiple plots?

[NPMSSupport]

A bit like the square importer, can plots be uploaded volume? If the organisation wishes to upload 5 plots for their 100 squares as an example, it would take a very long time to do them all manually.

[NPMSSupport]

@andrewvanbreda

[sacrevert]

It could be in theory, but not within the current resource.

[sacrevert]

@NPMSSupport Whilst I don't object to this in theory, I wonder how this will work in terms of how plots are created on the website? One obvious issue is the enabling of expert mode -- organisations may not have the GIS expertise to create a WKT boundary or line for a plot to include in an upload (presumably required to visualise a pre-imported plot in terms of a polygon or line on the website) -- ("Well Known Text", or WKT, is a way of describing a shape to be plotted in terms of a text string).

Also, people won't presumably know other details for the plots at the point of upload (e.g. slope, angle, SW corner) or have sketch maps, so plots will still require editing once on the website. Given that, as per the NPMS site, some aspects of plots can't be edited once samples have been recorded, this opens up a fair bit of scope for plots not to really have much descriptive information associated with them (and for lots of empty/unused plots to be created on the system).

Also noting here that this strategy goes against the NPMS guidance in terms of selecting plots in the field (although I appreciate that we are not currently providing organisations with the ability to generate maps with the preselected plot locations featured).

[andrewvanbreda]

I agree, there would be oddities with this using an importer I hadn't thought of. Like there would need to be an "Expert Mode" column to store the mode saved against the plot. The other trouble is how to we validate what is being imported is a valid combination.

[andrewvanbreda]

Note: Standard mode to be considered also (because of closure of #4)

[NPMSSupport]

@sacrevert very good points raised. I guess in terms of mapping skills of the user, could multiple plots be entered on purely a centroid GR and then the volunteer edits the positioning of it after taking on the square? This way the organisation still has the control over where the plot is, but the volunteer has to confirm the details (including SW corner, slope etc) before any data is added. Almost like pre populating the square with plots that are not fully finished, but still give the volunteer more of a start and they just edit the plots.

[sacrevert]

Yes that could work. We could also change it so that plots can always be edited. Whilst we have generally taken the view that this wouldn't be desirable for core NPMS plots, for smaller projects in NPMS+, it's not really an issue for us as the partnership.

@andrewvanbreda can you comment on, this, and maybe also any validation requirements on upload, and then update any quote for Sarah? Many thanks

[andrewvanbreda]

@sacrevert @NPMSSupport Yes I suppose this could work if we block off the option of selecting the other plots. I think we would need to remove any ability for the user to map to specific attributes during import process. There wouldn't need to really need to be any validation as we are just importing spatial reference effectively.

My main concern is how to make the importer understand the square it is importing into. I would need to check this.

[andrewvanbreda]

@sacrevert @NPMSSupport Have coded the technical element of this. Is this to be used by both public and admins? Currently the plan is to put button on the list plots page, but that page is available to both public and admins at moment, so if you need it for just admins some further coding will be required.

[sacrevert]

@andrewvanbreda presumably you mean project admins rather than Drupal admins here? If that's the case, then I don't think there is a big issue with project members also having access to an upload plots tool tbh

[andrewvanbreda]

@sacrevert Yes in this case would be project admins.

Righto, I will make available to members too. That is technically easier anyway.

[andrewvanbreda]

@sacrevert @NPMSSupport This is coded and the page is on the site (not in menu system yet). However I have had to also fix a bug at the warehouse end which is awaiting deployment. I guess that will be deployed Monday (but as am not doing that bit, I can't say for sure).

[andrewvanbreda]

@sacrevert @NPMSSupport The code has been copied to the warehouse, so I am doing final checks.

An issue I noticed is it isn't liking the import of the slope field, the terms for this are Flat (0-5°) Moderate (6-30°)
Steep (>30°)
Very steep to vertical (>45°)

I think it is the degrees character causing the issue. I don't think the problem is actually related directly to the importer, it is the windows/Linux file system that is moaning. Could we change these terms to remove that character? or change it to the word "degrees". It might also make it easier to do an import, as most people probably won't know how to find that character on their keyboard as the terms in the import file need to match what is on the Warehouse.

[sacrevert]

@andrewvanbreda Thanks Andy. Yes, good point. Please replace with degrees e.g. "Flat (0-5 degrees)" etc.

[andrewvanbreda]

Hi again @sacrevert Just to note, I have removed the default to "Flat" on the slope attribute on the Add Plot page (so affecting the main site, not importer). I have noticed the following unwanted behaviour. As it is not a required field, if a plot is imported without a Slope value, then if/when the user edits the plot on the website, Slope will default to flat during that edit, and the user might save the plot without noticing the automatic setting.

I will raise that behaviour in general Indicia issues, as I think it would be better to avoid default in edit mode even if no value.

NPMS is unaffected as doesn't have a plot importer, so have left that to default to Flat.

[andrewvanbreda]

@sacrevert Still not happy. But I can see the problem. It is moaning about the > symbol. This is because in Linux (Unix), this has meaning, it is used for directing data around in commands.

Can we change to the following? (note changes also affect NPMS) Flat (0-5 degrees) Moderate (6-30 degrees) Steep (greater than 30 degrees) Very steep to vertical (greater than 45 degrees)

[sacrevert]

Fine. Thanks Andy! Although this might be clearer: Flat (0-5 degrees) Moderate (6-30 degrees) Steep (31-45 degrees) Very steep to vertical (greater than 45 degrees)

[andrewvanbreda]

@NPMSSupport @sacrevert I have put a plot importer onto the Add Plots page. But this will do the import on a per square basis. (so for instance all the plots for 1 square can be done in one go) But looking back on the requirement I see you need this to work with the square indicated in the file itself. I need to think about that again, as there is an issue with that, as squares names are not unique across projects, and the importer won't understand that at the moment.

[sacrevert]

Can you include a column for project name too that is used to make sure that they are only imported to the stated project/square combination?

[andrewvanbreda]

@sacrevert That is essentially where the problem actually is unfortunately. The original theory was that the importer would use the project as a parameter in the URL (so really equivalent to what you are suggesting). However what it does at the moment is ignore the project (what Indicia calls a Recording Group) and just attaches to the first square with that name on the website. This logic is actually handled at the Warehouse end.

Then I thought we can just access the Plot Importer from the square, this doesn't have the issue as we attach directly onto square ID in the URL. However I did this without thinking the plots would be atatched to multiple squares in one batch. So that is what is live now.

So my recommendation.

1. Put full an importer live (it is easy to do this ignoring the bug), I don't think there are any duplicate squares at moment, and even if there was, there is a chance it will get it right, as it picks the first square. I think it will be painfully obvious if a square is wrong and can fix easily enough.

2. I will fix the bug in the Warehouse (as part of the Plantlife contract). After this is live sql can be easily run to detect if there were any previous issues.

3. If that fails we can do something to make the location code unique accross projects and use that as the identifier.

Andy

[andrewvanbreda]

@sacrevert I have been looking at the system. On option actually is that we can user the Indicia location ID as the parent square identifier. This would actually be most accurate and can be implemented now. The number can be added as a column to the squares list grid so people can see it. We obviously don't need to call it an Indicia ID number, we can label it as something else.

[sacrevert]

Ok, so that implies that a project admin can easily download a lot of their project squares with the indicia ID numbers clearly labelled. Is that the case currently?

[andrewvanbreda]

@sacrevert It isn't. But is ultra easy to add to Squares Administration grid, so they would see both the name and ID number next to each other.

[andrewvanbreda]

@sacrevert If that is confusing, I think we could make the importer use both name and ID number. So you map two columns. It may give peace of mind that the IDS aren't wrong. This combination could be clearer for a user.

[sacrevert]

both sounds sensible. Thanks Andy

[andrewvanbreda]

@sacrevert @NPMSSupport I have put this live as a button at the bottom of the Squares Administration page.

This should be good enough to put live, but I think I will try to fix some issues with the warehouse side of the importer we mentioned. I noticed another difficulty, when using the name and ID to indicate the parent, my tests showed that this did not perform how we would want. For instance, if the name was wrong but ID right (or vice versa), this would not be picked up as a validation problem.

So what I have done is just put it live where you specify the ID number as the parent only for now.

I will try and improve the warehouse side, as I think I still have some time available on the contract. Preferably, i would like to fix that issues so we can specify the project, we could then use the Square name instead of ID.

Anyway, it is live in a usable state now. There is also a Plot Importer for individual squares available on the Add Plots page.

[sacrevert]

@andrewvanbreda Yes, I fully agree that ideally we would have joint validation of square name and ID, otherwise it will be far harder for users to spot errors.

[andrewvanbreda]

Hi @sacrevert I will try to fix the other bug first which would be to allow it to recognise the group correctly, then we won't even need the ID. But can't do that now as doing stuff for David. I hope the current way the import is is good enough for any urgent imports.

[andrewvanbreda]

@sacrevert @NPMSSupport I have coded a way to fix this problem. It is the two column method, where both the Square Name and ID will be specified by the user, but the system will raise an error if those 2 columns are inconsistent. Before, if the Square Name was wrong for the specific ID, it would go ahead and attach the plot to the Square ID anyway. That is why the Square Name is currently hidden from view and only the Square ID can currently be used. This will prevent mistakes in the import file. I have had to fix this at the Warehouse end though, so I am not sure the exact timescale it will become available. Hopefully it won't be too long for the update to be applied to the Warehouse.