Open mccalluc opened 8 years ago
@mccalluc I'll investigate.
I tested a workflow that ran data through TextWrangler, using its "zap gremlins" function. This would be done to the XML payload bound for the ingestion process using Chuck's code, with options to process (or not) these things: Non-ASCII characters Control characters NULL (ASCII 0) characters to become either ASCII equivalent or HTML entity, e.g., Ò
@foo4thought Can you do this to the data and have the corrections make their way back to Filemaker itself? I'd rather the workflow be able to go from Database -> export-> server
Before we get too far into this, I can't swear that I'm not getting it wrong on my end. If it is upstream, I'd be a little concerned about:
to become either ASCII equivalent or HTML entity, e.g., Ò
That might not be sufficiently general: Consider a Vietnamese character with diacritics: Not going to have a named entity, I don't think, but we don't want to just strip diacritics either.
Well, the character is still present but not displaying on the current OV record page. Is it showing up on the actual page in the staging server? It doesn't show up in the Filemaker field for that record, only when pasting into a text editor.
Should we still try and process the actual encoding issues?
Here's a better diagnosis: The character in question is U+0092, which in Windows-1252 is an apostrophe. Most of Windows-1252 is identical to Latin-1, and Latin-1 in turn can be read as utf-8... but x80-x9F in 1252 is an exception: Those points are not in Latin-1, and are private use in Unicode.
@Muraszko : just do a search and replace in FM? Might also look for other characters in this range while you're on it? ... but if that's hard for some reason, not impossible to do adhoc cleaning downstream, though it wouldn't be my preference.
@mccalluc Thanks Chuck. I agree, clean it at the source.
@Muraszko : If you want another example: V_9E5411CEBD4447CE9D9DACFCAFAC0667
protesting against the forced labor of their husbands in Ngo Dinh Diems agroville or Strategic Hamlet program
There is a x93 / x94 pair on either side "Strategic Hamlet".
Thanks Chuck. I'll try and correct them in Filemaker.
For example: /catalog/V_2758B13C7014459498315BDF568EC3BC
Character encoding issue. See if the source data is correct, and possibly pass to Kevin, or maybe do an adhoc fix on this end.