Invalid unicode character sequences were found that could cause display issues on Apple News:

[parachute789]

The following JSON errors were detected: Invalid unicode character sequences were found that could cause display issues on Apple News. Stories in the cache wont publish because of this. I'm using the apple news plug in on wordpress an I have change the formatting. Could reseting it to normal help? If so how do I do that?

[bcampeau]

Can you please provide the JSON for the affected post here as code? Thanks.

[parachute789]

Hi,

This is all that I am getting as a notification. No code was available.

On Thu, Jan 12, 2017 at 3:31 PM, Bradford Campeau-Laurion < notifications@github.com> wrote:

Can you please provide the JSON for the affected post here as code? Thanks.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/alleyinteractive/apple-news/issues/271#issuecomment-272317431, or mute the thread https://github.com/notifications/unsubscribe-auth/AX3HIgCdwBIBv6x2k-X951kTChvDnyx4ks5rRre2gaJpZM4Le5cQ .

--

Annie Clark

Content Intern

parachutehome.com

headquarters 310.314.0119

mobile 562.881.3187

social @parachutehome

stay The Parachute Hotel http://theparachutehotel.com/

visit Showroom Store http://parachutehome.com/pages/showroom

[bcampeau]

Please follow the instructions in the wiki for exporting the JSON for this post. Thanks.

[parachute789]

Thanks for the instructions. Here is the code.

On Thu, Jan 12, 2017 at 3:39 PM, Bradford Campeau-Laurion < notifications@github.com> wrote:

Please follow the instructions in the wiki https://github.com/alleyinteractive/apple-news/wiki/usage#downloading-posts for exporting the JSON for this post. Thanks.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/alleyinteractive/apple-news/issues/271#issuecomment-272319018, or mute the thread https://github.com/notifications/unsubscribe-auth/AX3HIgC5UBZpz-hWnWBqWzyN8-WWKAqgks5rRrm_gaJpZM4Le5cQ .

--

Annie Clark

Content Intern

parachutehome.com

headquarters 310.314.0119

mobile 562.881.3187

social @parachutehome

stay The Parachute Hotel http://theparachutehotel.com/

visit Showroom Store http://parachutehome.com/pages/showroom

[bcampeau]

I don't see anything posted here. Alternatively you can use a github gist and link to that.

Thanks.

[parachute789]

{"version":"1.1","identifier":"post-6503","language":"en","title":"How to Use Crystals in Your Home","documentStyle":{"backgroundColor":"#fafafa"},"layout":{"columns":7,"width":1024,"margin":100,"gutter":20},"components":[{"role":"header","layout":"headerPhotoLayout","components":[{"role":"photo","layout":"headerPhotoLayout","URL":"bundle:\/\/How-to-Use-Crystals-in-Your-Home_Hero_EnergyMuse-e1475715574690.jpg"}],"behavior":{"type":"parallax","factor":0.8}},{"role":"container","layout":{"columnStart":0,"columnSpan":7,"ignoreDocumentMargin":true},"style":{"backgroundColor":"#fafafa"},"components":[{"role":"title","text":"How to Use Crystals in Your Home","textStyle":"default-title","layout":"title-layout"},{"role":"byline","text":"by Christine Diaz | Oct 4, 2016 | 6:33 PM","textStyle":"default-byline","layout":"byline-layout"},{"role":"body","text":"For centuries, crystals have been used in homes to ward off negative feelings, encourage prosperity, improve sleep and create harmony. Even the most common crystals on the market work on a vibrational level and have the ability to heal, protect and transmute energy. Your home is where you unwind and connect with loved ones \u2013 and crystals can help ensure your space remains both cozy and safe. \n\nIf you\u2019re just getting started using crystals, choosing the right one can be overwhelming\u00a0\u2013\u00a0what exactly are they, anyway? We turned to Ashley Neese, a Los Angeles-based energy healer and certified breathwork teacher, to explain how they work. Here\u2019s\u00a0how to integrate\u00a0them into your life \u2013\u00a0starting with your home.","format":"markdown","textStyle":"dropcapBodyStyle","layout":"body-layout"},{"role":"heading3","text":"Crystals: What Are They?","format":"markdown","textStyle":"default-heading-3","layout":"heading-layout"},{"role":"photo","URL":"bundle:\/\/How-to-Use-Crystals-in-Your-Home_2_Retouched.jpg","layout":"anchor-layout-right","anchor":{"targetAnchorPosition":"center","rangeStart":0,"rangeLength":1,"targetComponentIdentifier":"component-c23c3162582743a640c3dc3ef01d9b9d"}},{"role":"body","text":"Crystals can serve many functions in your home space. Source: Asami Zenri\/Ashley Neese\n\nCrystals are formed when minerals bond together in a specific pattern \u2013 this pattern is what determines the type of crystal (not the\u00a0outer appearance). The combination of heat and the pressure underground forms crystals from\u00a0liquid minerals \u2013 a process that often takes millions of years. We like to think about\u00a0crystals as the Earth\u2019s DNA because they contain\u00a0the chemical imprints of the geological evolution that created their structures. In some native cultures, crystals are referred to as \u201cMother Earth\u2019s brain cells,\u201d because the changes a crystal goes\u00a0through is\u00a0recorded in its\u00a0makeup!","format":"markdown","textStyle":"default-body","layout":"body-layout","identifier":"component-c23c3162582743a640c3dc3ef01d9b9d"},{"role":"heading3","text":"Four Starter Crystals for the Home","format":"markdown","textStyle":"default-heading-3","layout":"heading-layout"},{"role":"photo","URL":"bundle:\/\/quad-4-860x1024.jpg","layout":"anchor-layout-right","anchor":{"targetAnchorPosition":"center","rangeStart":0,"rangeLength":1,"targetComponentIdentifier":"component-5a55ebc4637ed9147224cd5c9cf93a27"}},{"role":"body","text":"Clockwise: Black Tourmaline, Source: The Hippie Ship\/Etsy; Citrine, Source: Gypsy Sue Designs\/Etsy; Rose Quartz, Source: Midnight In Seattle\/Etsy; Shungite, Source The Boho Cat Shop\/Etsy\n\nEach crystal has a specific vibration\u00a0to absorb and transform energy, so\u00a0you should place\u00a0specific crystals in different areas of your home to meet the unique\u00a0needs of the\u00a0space. Here are four of\u00a0the most versatile crystals to get you started:\n\nBlack Tourmaline:\u00a0One of the most grounding crystals, Black Tourmaline protects against all forms for negative energy, geopathic stress and EMFs. To protect your home, make a\u00a0grid by placing a a piece of Black Tourmaline in each of the main corners of either the building or the property outside. If you want extra protection in a specific space, place\u00a0a crystal in every corner of that room. When creating a grid, connect each crystal with the same intention during activation (more on that below). You should say something like, \u201cMay this home and everyone within be protected at all times.\u201d\n\nCitrine:\u00a0If you desire to manifest abundance and prosperity, Citrine is a must. Place this yellowish quartz in the wealth corner \u2013 the\u00a0southeast corner \u2013 of your home or office. Citrine is sometimes\u00a0called \u201cthe merchant\u2019s stone\u201d because it has historically been used for success around work, and it radiates energy to everyone involved. Be wary when purchasing Citrine because dishonest dealers occasionally heat treat Amethyst, a less expensive crystal, to make it look like Citrine.\n\nRose Quartz:\u00a0Create a nurturing and safe retreat in your bedroom by placing several Rose Quartz crystals throughout the space to establish an atmosphere of gentle kindness. Rose Quartz activates and balances the heart center and represents the vibration of calming, soothing love. Because it encourages rest and sweet dreams, it is a wonderful crystal for children\u2019s rooms.\n\nShungite:\u00a0Modern homes are overloaded with harmful energy from EMFs (electromagnetic frequencies),\u00a0emitted\u00a0by computers and smartphones. Shungite\u2019s special properties have been proven to neutralize and purify EMFs (for science buffs, check out the 1996 Nobel Prize in Chemistry). Place\u00a0a piece of Shungite on your desk to\u00a0transform electromagnetic build up and keep your workspace clear. You should\u00a0also put smaller pieces of Shungite near outlets when you are charging electronics.","format":"markdown","textStyle":"default-body","layout":"body-layout","identifier":"component-5a55ebc4637ed9147224cd5c9cf93a27"},{"role":"heading3","text":"Choosing\u00a0the Right Crystal","format":"markdown","textStyle":"default-heading-3","layout":"heading-layout"},{"role":"photo","URL":"bundle:\/\/How-to-Use-Crystals-in-Your-Home_1_Retouched.jpg","layout":"anchor-layout-right","anchor":{"targetAnchorPosition":"center","rangeStart":0,"rangeLength":1,"targetComponentIdentifier":"component-f40a6648d8c052ab86802fedf8286c9a"}},{"role":"body","text":"Select a\u00a0crystal that brings you joy\u00a0and that supports\u00a0your lifestyle; Source: Asami Zenri\/Ashley Neese\n\nWhen purchasing crystals, consider the source. Always ask questions at your local crystal shop or spend time researching reputable\u00a0online stores. Unfortunately,\u00a0unethical mining practices are common, and crystals absorb this negative energy.\u00a0We recommend attending a\u00a0gem or mineral show if possible, as you will be able to meet directly with the buyers and miners to\u00a0gather\u00a0valuable\u00a0information about their mining practices.\n\nOften times, you will find yourself drawn to a particular crystal. When that happens, go with it! Pick it up. Connect to its vibration. If it feels right, take it\u00a0home and research its benefits.\u00a0If choosing a crystal from the list above, handle a couple of them (or browse\u00a0several\u00a0options if buying online) and pick the one that you feel most connected with. You cannot make a wrong decision. When in doubt, go with your gut.","format":"markdown","textStyle":"default-body","layout":"body-layout","identifier":"component-f40a6648d8c052ab86802fedf8286c9a"},{"role":"heading3","text":"Cleansing and Activating Your Crystal","format":"markdown","textStyle":"default-heading-3","layout":"heading-layout"},{"role":"body","text":"After you acquire a new crystal, it\u2019s necessary to give it\u00a0an energy cleanse \u2013 a simple dip in salt water is the easiest method for Black Tourmaline, Rose Quartz and Shungite.\n\nHowever, some crystals are damaged by water (a general rule of thumb, don\u2019t use water to cleanse a stone under 7 on\u00a0Mohs scale of mineral hardness). Cleanse these\u00a0by exposure to the light of a full moon. Bring the crystals inside a few hours after sunrise. Another alternative is to immerse\u00a0them in palo santo smoke.\u00a0We\u00a0recommend you cleanse your\u00a0crystals once a\u00a0month, as they are constantly soaking up energy.\u00a0Note that Citrine (and a\u00a0few other crystals) does not require\u00a0cleansing.\n\nOnce it\u00a0is cleansed, infuse\u00a0your intention into the crystal to activate it. Do this by taking the crystal in your hand, slowing down your breathing, focusing on your intention and then meditating.\u00a0If you are activating your crystal to perform a specific function (e.g. attract prosperity in your career), be as specific with the wording of your intention as possible. Repeat your intention out loud a few times, then\u00a0close the activation with a long exhale and gratitude for the crystal\u2019s support. Remember that each time you cleanse your crystal, it\u00a0needs to be reactivated.","format":"markdown","textStyle":"default-body","layout":"body-layout-last"}]}],"componentTextStyles":{"dropcapBodyStyle":{"textAlignment":"left","fontName":"AvenirNext-Regular","fontSize":18,"lineHeight":24,"textColor":"#4f4f4f","linkStyle":{"textColor":"#428bca"},"paragraphSpacingBefore":18,"paragraphSpacingAfter":18,"dropCapStyle":{"numberOfLines":4,"numberOfCharacters":1,"padding":5,"fontName":"AvenirNext-Bold","textColor":"#4f4f4f"}},"default-body":{"textAlignment":"left","fontName":"AvenirNext-Regular","fontSize":18,"lineHeight":24,"textColor":"#4f4f4f","linkStyle":{"textColor":"#428bca"},"paragraphSpacingBefore":18,"paragraphSpacingAfter":18},"default-heading-3":{"fontName":"AvenirNext-Bold","fontSize":18,"lineHeight":48,"textColor":"#333333","textAlignment":"left"},"default-title":{"fontName":"AvenirNext-Bold","fontSize":48,"lineHeight":48,"textColor":"#333333","textAlignment":"left"},"default-byline":{"textAlignment":"left","fontName":"AvenirNext-Regular","fontSize":14,"textColor":"#7c7c7c"}},"componentLayouts":{"body-layout":{"columnStart":0,"columnSpan":5,"margin":{"top":12,"bottom":12}},"body-layout-last":{"columnStart":0,"columnSpan":5,"margin":{"top":12,"bottom":30}},"heading-layout":{"columnStart":0,"columnSpan":7,"margin":{"top":15,"bottom":15}},"anchored-image":{"margin":{"top":25,"bottom":25}},"anchor-layout-right":{"columnStart":3,"columnSpan":4},"headerPhotoLayout":{"columnStart":0,"columnSpan":7,"ignoreDocumentMargin":true},"headerBelowTextPhotoLayout":{"columnStart":0,"columnSpan":7,"ignoreDocumentMargin":true,"margin":{"top":30,"bottom":0}},"title-layout":{"margin":{"top":30,"bottom":0}},"byline-layout":{"columnStart":0,"columnSpan":7,"margin":{"top":10,"bottom":10}}},"metadata":{"excerpt":"Choose the right crystals for your home with the guidance of an energy healer.","thumbnailURL":"bundle:\/\/How-to-Use-Crystals-in-Your-Home_Hero_EnergyMuse-e1475715574690.jpg","dateCreated":"2016-10-05T01:33:12+00:00","dateModified":"2016-10-06T21:37:24+00:00","datePublished":"2016-10-05T01:33:12+00:00","canonicalURL":"http:\/\/ blog.parachutehome.com\/four-crystals-for-home\/","generatorIdentifier":"publish-to-apple-news","generatorName":"Publish to Apple News","generatorVersion":"1.2.0"},"advertisingSettings":{"frequency":1,"layout":{"margin":{"top":15,"bottom":15}}}}

On Thu, Jan 12, 2017 at 3:55 PM, Bradford Campeau-Laurion < notifications@github.com> wrote:

I don't see anything posted here. Alternatively you can use a github gist https://gist.github.com and link to that.

Thanks.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/alleyinteractive/apple-news/issues/271#issuecomment-272321677, or mute the thread https://github.com/notifications/unsubscribe-auth/AX3HIglnYxgCgWXaTyhz_GukEvlO3oBSks5rRr1dgaJpZM4Le5cQ .

--

Annie Clark

Content Intern

parachutehome.com

headquarters 310.314.0119

mobile 562.881.3187

social @parachutehome

stay The Parachute Hotel http://theparachutehotel.com/

visit Showroom Store http://parachutehome.com/pages/showroom

[kevinfodness]

Hi @parachute789—

I'm having a bit of trouble with the JSON you included. I believe the formatting was corrupted when it was pasted. Can you either paste the JSON as code or link to a GitHub gist please?

This issue can happen if content was imported into WordPress improperly, which may happen if you are pasting content in from a different program. Have you encountered this issue when writing content natively in WordPress?

[AaronHolbrook]

I am having this same exact issue AFTER updating to your latest version: 1.2.2

I believe you need the JSON_UNESCAPED_UNICODE flag on the json_encode() call: http://php.net/manual/en/function.json-encode.php

Here are two gists, one WITH the flag, and one WITHOUT:

https://gist.github.com/AaronHolbrook/934b2fd8109e20eaa78fbe5daff130c5

This is the fix (instead of trying to do the preg_match to replace it, which did not work:

[AaronHolbrook]

And for the record, the content that I have in the DB is clean, there is absolutely nothing I can identify that is wrong with it:

Anti-Donald Trump protesters temporarily shut down a highway in New Haven, Conn. on Saturday, forcing ambulance workers who were delayed by the demonstration to perform an emergency medical procedure in the vehicle.

[AaronHolbrook]

here is the article in question in apple news: - notice the issue after the word: temporarily

[AaronHolbrook]

Actually that didn't work as well as I had hoped.

I think I may have stumbled across something: it appears that unicode hides &nbsp characters.

http://stackoverflow.com/questions/9870974/replace-nbsp-characters-that-are-hidden-in-text

Going to test out a fix going in this direction.

[AaronHolbrook]

You marked this as closed, but it actually made the issue worse:

Whilst on 1.2.2:

Upgraded to 1.2.4:

[kevinfodness]

Hi @AaronHolbrook—

I'll reopen the issue and try to get to the bottom of your particular use case. Other reporters have come back saying that the fix solved their problem, which means that something else is going on in your case.

Can you please send the raw post content (copied from the text tab) via a pastebin.com link so that I can reproduce locally?

Thanks.

[AaronHolbrook]

Sure @kevinfodness I am also trying to get to the bottom of this. It appears in the chrome inspector as a  :

However when viewing the actual source of the file, that is not the case:

https://gist.github.com/AaronHolbrook/1d70756c5e568205fd853004a4a33582

[AaronHolbrook]

Here is a possibly a better export of the bad/malformed content: http://d.pr/f/nPy6/28TrMKW2

[AaronHolbrook]

There is a supposed   between the words country and while

[AaronHolbrook]

I'm seeing the UTF-8 hex: C2 A0 which looks like it is a 'no-break space' or U+00A0

[AaronHolbrook]

Is it possible to add a filter that people can intercept to clean things like this up themselves if all else fails?

[kevinfodness]

The latest release cleans up special Unicode spacing characters and converts them to standard spaces.

We have a filter for the generated JSON already, called apple_news_generate_json: https://github.com/alleyinteractive/apple-news/wiki/Action-and-Filter-Hooks#filter-hooks

I'm looking at the second link to content you provided. The first one rendered fine.

[kevinfodness]

I can't reproduce this issue on my end. I've tried loading in your example content using several different mechanisms, including manually inserting an   at the location you specified, tabbing to the visual editor to render it into UTF-8, and then exporting. Using the latest version of the plugin (1.2.4) everything is rendering as expected in the News Preview tool.

For reference, I'm using the following:

WordPress 4.7.3 Publish to Apple News 1.2.4 PHP 7.0.15 MariaDB 10.0.29

[AaronHolbrook]

Yea, I can't reproduce this locally, your change in the latest version seems to resolve the issue locally only.

The production environment unfortunately is where it is hitting a snag.

It is an older version of PHP and Apache, which would be the difference and could be the issue.

Thanks for your assistance on this.

On Mar 7, 2017, 1:44 PM -0600, Kevin Fodness notifications@github.com, wrote:

I can't reproduce this issue on my end. I've tried loading in your example content using several different mechanisms, including manually inserting an   at the location you specified, tabbing to the visual editor to render it into UTF-8, and then exporting. Using the latest version of the plugin (1.2.4) everything is rendering as expected in the News Preview tool. For reference, I'm using the following: WordPress 4.7.3 Publish to Apple News 1.2.4 PHP 7.0.15 MariaDB 10.0.29 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[kevinfodness]

That's good to know. What version of PHP is the production server running?

The relevant bits of code that are intended to fix the Unicode issues are here:

https://github.com/alleyinteractive/apple-news/blob/master/includes/apple-exporter/class-exporter.php#L74-L92

and here:

https://github.com/alleyinteractive/apple-news/blob/master/includes/apple-exporter/class-exporter.php#L355-L361

I would guess that the part that is failing is the json_decode lines that convert the Unicode escape sequences, expressed in JavaScript notation, into characters in the local Unicode encoding. Would it be possible to run a test on the production server to determine if that method of decoding Unicode escape sequences is executing properly?

[AaronHolbrook]

That was my thinking also.

PHP version 5.4.45

json support    enabled
json version    1.2.1

Yea, I can definitely run a test online, did you have something specific in mind on how to test?

[AaronHolbrook]

For what it's worth, I was able to do the string replacement without involving json_decode in the following manner:

str_replace( "\xC2\xA0", "ZZZZZZZZZ", $data )

Reference: http://stackoverflow.com/questions/3959626/replace-unicode-character

[kevinfodness]

Try this as test code:

<?php

$a = json_decode( '"\u00a0"' );
$b = html_entity_decode( '&nbsp;' );
var_dump( $a === $b );

I've verified this works properly on PHP 7.0.15 and 5.5.38.

I looked into the method you mentioned, but the specific escape sequences that are required when using that method vary depending on the settings on individual systems. For example, it's \xC2\xA0 in UTF-8, but \x00\xA0 in UTF-16, etc. Using json_decode was intended to standardize the transformation process to allow PHP to account for system differences in encoding.

Ref: http://www.fileformat.info/info/unicode/char/00a0/index.htm

[AaronHolbrook]

Returns true on php 5.4.45.

[kevinfodness]

Can you try this version, which more closely approximates the actual operation?

<?php

$data = html_entity_decode( 'Test&nbsp;&ldquo;content&rdquo;' );
$separators = array(
    json_decode( '"\u0020"' ),
    json_decode( '"\u00a0"' ),
    json_decode( '"\u1680"' ),
    json_decode( '"\u2000"' ),
    json_decode( '"\u2001"' ),
    json_decode( '"\u2002"' ),
    json_decode( '"\u2003"' ),
    json_decode( '"\u2004"' ),
    json_decode( '"\u2005"' ),
    json_decode( '"\u2006"' ),
    json_decode( '"\u2007"' ),
    json_decode( '"\u2008"' ),
    json_decode( '"\u2009"' ),
    json_decode( '"\u200a"' ),
    json_decode( '"\u202f"' ),
    json_decode( '"\u205f"' ),
    json_decode( '"\u3000"' ),
);
$data = str_replace( $separators, ' ', $data );
$data = preg_replace( '/\h+/', ' ', $data );
var_dump( 'Test “content”' === $data );

[AaronHolbrook]

True still:

[AaronHolbrook]

In an attempt to more closely resemble the process, I added what happens after that during the class-exporter:

It appears it may be related to json_encode on PHP 5.4

[AaronHolbrook]

This is the output on PHP 5.4.45:

bool(true) string(26) ""Test \u201ccontent\u201d""

[AaronHolbrook]

Nevermind, I was seeing the \u201\ and thinking it was the separator, but it was in fact the curly quotes you had html decoded.

[kevinfodness]

Yep, that's expected behavior, so it looks like it's properly translating the non-breaking space into a regular space. Are you absolutely certain that your production site is running version 1.2.4 of Publish to Apple News? Can you try reinstalling a fresh version of the plugin to be sure?

[AaronHolbrook]

I am certain it is running 1.2.4.

I have hacked in some error logging:

Here is the output:

https://gist.github.com/AaronHolbrook/00276cf6a2cd1c9e804624744dcce3e9

It appears that even before the string replacement the unicode is off.

[AaronHolbrook]

Ok, unfortunately I am still unable to determine why the unicode that is coming through at that point is mangled; however, in the interest of fixing the original problem I am going with the following solution by using your filter and similar recursive approach, but adding additional string replacement:

add_filter( 'apple_news_generate_json', function( $json ) {

    $json = clean_array_recurse( $json );

    return $json;
} );

function clean_array_recurse( &$data ) {

    // If the value is an array, loop through it and process each element.
    if ( is_array( $data ) ) {
        foreach ( $data as &$datum ) {
            clean_array_recurse( $datum );
        }
    }

    // If the value is a string, clean it up.
    if ( is_string( $data ) ) {
        $data = str_replace( "\xc2\xA0", " ", $data );
        $data = str_replace( "\xc2\x82", "", $data );
        $data = str_replace( "\xc3\x82", "", $data );
        $data = str_replace( "\xc2\x9c", "", $data );
        $data = str_replace( "\xc3\x83", "", $data );
    }

    return $data;
}

[AaronHolbrook]

You can close this @kevinfodness; we'll chalk it up to some weird contextual server issue at this point and I can deal with any other inconsistencies with the above approach.

Many thanks for your help today!

[kevinfodness]

Glad to assist!