search

citrit / iMapInvasiveSpeciesApp

iMap invasive species app for field entry. iMapInvasives provides an on-line, GIS-based data management system to assist citizen scientists and natural resource managers working to protect natural resources from the threat of invasive species.
3 stars 0 forks source link

Investigate special characters in text fields #61

Closed jlmarino closed 5 years ago

jlmarino commented 6 years ago

From @citrit :

I am pretty sure the app encodes the strings prior to sending via http so we need an equivalent decode on the iMap server side.

John can take a look at the code and maybe we look to filter the comments prior to sending and replace it remove unknown characters.

shodge17 commented 6 years ago

I accidently added this comment to one of the other issues.

Here is another example of what ended up in the obscomments_long field: 'Abundance - Size of Area: Up to 0.5 acre (21,780 sq. ft)\nAbundance - Distribution: Dense plants/clumps\nAilanthus with DBH six inches or greater present: Yes\nGeneral Comments:\nInstallation of \xe2\x80\x9csticky tape\xe2\x80\x9d for SLF monitoring'

I can see many records that have comments that use your code and the above record and the previous one seem to be the only ones that have failed so far.

jlmarino commented 6 years ago

Thanks, @shodge17. Is there an obs record for this, or does iMap fail to create a record whenever these characters are present?

shodge17 commented 6 years ago

The record gets successfully created, but when you try to call it up online it blows up. Attached is the email I get from the system when this happens. nyimiobservationNY-449109U.txt

jlmarino commented 6 years ago

Thanks.

Also, a few hours ago, I uploaded two records into the dev site with special characters -- one had an emoji, and the other had special quotes and en dashes. Did both of these records also trigger the email alerts? Does it appear to be the same problem for each?

In the obs records on the web dev site, the record with the emoji character works, and the emoji is represented by four other characters, while the record with special quotes/en dashes record is showing a Django error page. The records are: NY-321359U and NY-321358U.

Hope this helps.

shodge17 commented 6 years ago

So I guess we need to figure out if the fix will be on your end or mine. I am going to rely on your team to help make that determination. Is it easy for you to strip that kind of stuff out before it is sent? If not I can look into how to fix it on my end.

citrit commented 6 years ago

It is easy to only accept certain inputs, I am not sure trying to define what to remove is a good idea. Instead we define what is accepted input and filter all comments to only accept what is determined safe. I agree the decision is to to Meg and team.

jlmarino commented 5 years ago

iMap3 now uses utf-8 – this issue should now be all set