search

textpattern / textpattern-plugins-website

Textpattern CMS Plugins website.
https://plugins.textpattern.com
GNU General Public License v2.0
7 stars 1 forks source link

Output JSON files for auto update process #10

Closed philwareham closed 4 years ago

philwareham commented 4 years ago

Stub issue to track the format we will (hopefully) use for Textpattern 4.9 core plugin auto-update system.

Currently, JSON template will aim to give the following data (TBC):

{
  "name": "{prefix_plugin-name}",
  "supersededByCore": {boolean},
  "beta": {
    "version": "{semver}",
    "verifiedMaxTxpCompatibility": "{ver-of-textpattern}",
    "datePublished": "{yyyy}-{mm}-{dd}",
    "endpointUrl": "{url-of-plugin-beta-endpoint}"
  },
  "stable": {
    "version": "{semver}",
    "verifiedMaxTxpCompatibility": "{ver-of-textpattern}",
    "datePublished": "{yyyy}-{mm}-{dd}",
    "endpointUrl": "{url-of-plugin-stable-endpoint}"
  }
}

For example (TBC):

{
  "name": "com_connect",
  "supersededByCore": 0,
  "beta": {
    "version": "4.6.1-beta.3",
    "verifiedMaxTxpCompatibility": "4.9",
    "datePublished": "2020-06-29",
    "endpointUrl": "https://github.com/textpattern/com_connect/archive/4.6.1-beta.3.zip"
  },
  "stable": {
    "version": "4.6.0",
    "verifiedMaxTxpCompatibility": "4.8",
    "datePublished": "2020-01-31",
    "endpointUrl": "https://github.com/textpattern/com_connect/archive/4.6.0.zip"
  }
}

The reason we don't show verifiedTxpCompatibility for beta releases, is that these are only shown when their semver is higher than stable (or no stable exists). I'm open to opinion on this. Also TBC whether we even show beta releases here at all (i.e. should we exempt beta releases from the auto-update system)?

philwareham commented 4 years ago

@bloke throwing this open to debate - all yours! :)

Bloke commented 4 years ago

Good question about betas. I'm of the opinion we allow beta updates. But we hide them by default and have some checkbox, either on the Plugins panel or as a pref, that allows you to consider betas/alphas or whatever as fair game. Does that work?

Also, could we utilise a more generic supersededBy? Then the value can be core or another plugin name (e.g. zem_contact_reborn is superseded by com_connect). That gives us scope to link to somewhere else; perhaps the plugin, or even a tutorial or something on TxpTips to explain how to replace abc_plugin with core tags?

{
  "name": "zem_contact_reborn",
  "supersededBy": {
      "name": "com_connect",
      "location": "https://github.com/textpattern/com_connect",
  },
...

or

{
  "name": "smd_macro",
  "supersededBy": {
      "name": "shortcodes",
      "location": "https://docs.textpattern.com/tags/shortcodes/",
  },
...
philwareham commented 4 years ago

Scratching my head about how we output this in Textpattern. Effectively we want to use an article (the plugin individual article) in two sections: section 1 being the standard web page, section 2 being the JSON output (kind of like a feed).

For example...

  1. https://plugins.textpattern.com/plugins/abc_example: the standard web page with info about the plugin.

2a. https://plugins.textpattern.com/plugins-json/abc_example: the JSON extended data about the plugin, or 2b. https://plugins.textpattern.com/plugins/abc_example/json: the JSON extended data about the plugin.

Can this be done via core or is it plugin territory? Am I missing something obvious?

Bloke commented 4 years ago

If you register a JSON mime type in Advanced settings, this should be doable from a form. That means the single plugin page can render both visible components and JSON components if you like. Or, perhaps better, we could use your 2b endpoint, which I quite like:

You can do that in any page or form - provided no output has been sent to the browser beforehand - with a little evaluate magic and txp:page_url's ability to extract numerical URL components. In this case we're testing the 3rd component is 'json', after 1=section, 2=title in our permlink scheme:

<txp:evaluate query='"<txp:page_url type="3" />" = "json"'>
   <txp:header value="application/json; charset=utf-8" />{
   ... rest of JSON
<txp:else />
   <html>
      <head>
      ... regular individual article flow
</txp:evaluate>
philwareham commented 4 years ago

Brilliant, that works perfectly!

If you register a JSON mime type in Advanced settings

What's this bit though (it seems to work already without me doing this)?

Bloke commented 4 years ago

Yeah, forget the registration. That was an alternative if you wanted to use <txp:output_form> and have it spit out the correct header on your behalf. Since you're using <txp:header> manually, there's no need for the registration.

philwareham commented 4 years ago

OK, plugin JSON files for hopeful use in the auto-update feature will reside in the following URL scheme

https://plugins.textpattern.com/plugins/abc_example/json

For example: https://plugins.textpattern.com/plugins/com_connect/json

I've not put most of the dynamic logic in yet, but it's a WIP.

Bloke commented 4 years ago

That is superb work, thank you.

Just been browsing the (rah-heavy!) collection, filtering stuff and so forth, sticking the odd /json on the end. That's gonna be great, and all the extra bits of info about a plugin and its compatibility are invaluable for people wanting to browse the repo and find what's available.

philwareham commented 4 years ago

@bloke I'm working on the JSON output of legacy nodes. I have this PHP code although I have a some questions:

if (!empty($json->legacy)) {
    echo '  "legacy": [';

    foreach ($json->legacy as $avail) {
        foreach ($avail as $txpver => $legacy) {
            $safe_txpver = txpspecialchars($txpver);
            $safe_plugver = txpspecialchars($legacy->version);

            echo n.'    {';
            echo n.'      "'.$safe_txpver.'" {';
            echo n.'        "version": "'.$safe_plugver.'",';

            if (!empty($legacy->datePublished)) {
                echo n.'        "datePublished": "'.txpspecialchars($legacy->datePublished).'",';
            }

            if (!empty($legacy->downloadUrlPhp)) {
                echo n.'        "endpointUrl": "'.txpspecialchars($legacy->downloadUrlPhp).'"';
            } else {
                echo n.'        "endpointUrl": "'.txpspecialchars($legacy->downloadUrlTxt).'"';
            }

            echo n.'      }';
            echo n.'    },';
        }
    }

    echo n.'  ],';
}

Question 1: How would I trim the final comma in the foreach ($avail as $txpver => $legacy) loop for the final node item in loop. Question 2: Should I put whitespaces in the echo output (this is mainly for readability, I know JSON whitespace is ignored), looks a bit hacky to me? Question 3: most importantly, is this PHP code actually really necessary as it effectively remakes the JSON already supplied from the original curated-plugins-list JSON files. Is there a cleaner way of grabbing that original slug of data and outputting?

There is a caveat on question 3, there is a slight different in the legacy JSON output from the site compared to the original curated-plugins-list JSON: the potential two download URLs downloadUrlPhp and downloadUrlTxt are a singleendpointUrl where the downloadUrlPhp value is used as preference, but if missing uses downloadUrlTxt value instead.

Hope that makes some sort of sense?

Bloke commented 4 years ago

The answer to all three questions is to use json_decode and json_encode.

If you read in the original file with json_decode($string_from_get_file_contents, true); it becomes a bog standard array. You can simply add elements to it or remove others by using unset().

Then just json_encode($json) it back up again and squirt it out.

philwareham commented 4 years ago

Sorry, can you provide example code again! I have no PHP knowledge whatsoever so I just fudge it which takes quite a bit of time. Apologies for keeping you busy - it'll be worth it!

Bloke commented 4 years ago

Sure, something like this:

$url = 'https://plugins.textpattern.com/plugins/rah_pathway/json';
$json = file_get_contents($url);
$json = json_decode($json, true);

// Create a legacy node from one of the other two.
if (!empty($json['legacy']['downloadUrlPhp'])) {
    $json['legacy']['endpointUrl'] = txpspecialchars($json['legacy']['downloadUrlPhp']);
} elseif (!empty($json['legacy']['downloadUrlTxt'])) {
    $json['legacy']['endpointUrl'] = txpspecialchars($json['legacy']['downloadUrlTxt']);
}

// Remove entries we don't need any more.
unset(
    $json['legacy']['downloadUrlPhp'],
    $json['legacy']['downloadUrlTxt']
);

// Bundle the array back up and write it to file.
$json = json_encode($json);
file_put_contents('/path/to/new/json', $json);
philwareham commented 4 years ago

@bloke I only need to extract the legacy nodes from the original JSON file, iterate over it to create any endpointUrls and remove downloadUrlPhp and downloadUrlPhp - then add that to my new JSON file. The rest of the new JSON file we are building from information available in the plugins site directly.

I think you example above takes the whole original JSON file and squirts that all back out with the legacy amends in place?

This is the last problem to solve for plugin pages. After that it's just getting search/version filters working on homepage and we can put this live for testing.

Bloke commented 4 years ago

Yeah, if you're removing the content, just make a new variable to hold the new JSON file:

$url = 'https://plugins.textpattern.com/plugins/rah_pathway/json';
$json = file_get_contents($url);
$json = json_decode($json, true);
$newJson = array();

// Create a legacy node from one of the other two.
if (!empty($json['legacy']['downloadUrlPhp'])) {
    $newJson['legacy']['endpointUrl'] = txpspecialchars($json['legacy']['downloadUrlPhp']);
} elseif (!empty($json['legacy']['downloadUrlTxt'])) {
    $newJson['legacy']['endpointUrl'] = txpspecialchars($json['legacy']['downloadUrlTxt']);
}

// Remove entries we don't need any more if you intend to write the old file back,
// otherwise just delete this bit.
unset(
    $json['legacy']['downloadUrlPhp'],
    $json['legacy']['downloadUrlTxt']
);

// Write the current JSON file out with removed nodes, if necessary.
// Delete this bit otherwise.
$json = json_encode($json);
file_put_contents('/path/to/current/file.json', $json);

// Bundle the new array back up and write it to file.
$out = json_encode($newJson);
file_put_contents('/path/to/new/file.json', $out);
philwareham commented 4 years ago

Hmmm, I can't seem to get this working. Is this correct...

['legacy']['downloadUrlPhp']

...as isn't there a node in between them kind of like...

['legacy']['the-txp-version']['downloadUrlPhp']
philwareham commented 4 years ago

@bloke sorry to bump this, last issue I need to get fixed on the plugins pages now.

Bloke commented 4 years ago

Ah yeah, you'll need an extra layer of iteration to get the Txp version. How do you want to write this out in the final JSON file? Do you want to keep the Txp version in the hierarchy?

"legacy": {
    "4.6.0": {
        "endpointUrl": "https://github.com/textpattern/com_connect/archive/4.6.0.zip"
    }
}

OR:

"legacy": {
    "txp-version": "4.6.0",
    "endpointUrl": "https://github.com/textpattern/com_connect/archive/4.6.0.zip"
  }
philwareham commented 4 years ago

Like this:

  "legacy": [
    {
      "4.6": {
        "version": "v4.5.1",
        "datePublished": "2016-12-03",
        "endpointUrl": "https://github.com/textpattern/com_connect/archive/v4.5.1.tar.gz"
      }
    },
    {
      "4.5": {
        "version": "v4.5.0.0-beta.4",
        "datePublished": "2016-12-02",
        "endpointUrl": "https://github.com/textpattern/com_connect/archive/v4.5.0.0-beta.4.tar.gz"
      }
    }
  ],
Bloke commented 4 years ago

Cool. Got any plugins I can test it on? https://plugins.textpattern.com/plugins/com_connect/json is currently spewing out:

SyntaxError: JSON.parse: expected ':' after property name in object at line 11 column 13 of the JSON data

("4.5" and "4.6" need colons after them)

philwareham commented 4 years ago

Just fixed that temporarily, but really we need to sort out the legacy output as discussed (grabbing the legacy JSON from original file, amending the endpoint, and re-encoding). Currently there is an erroneous extra comma in my output, using my dirty code method.

Bloke commented 4 years ago

Oh, duh, I was supposed to be getting the original from GitHub, not the repo, sorry. Anyway, try this:

// Create legacy node from one of the endpoints.
if (!empty($json['legacy'])) {
    foreach ($json['legacy'] as $txpBlock) {
        foreach ($txpBlock as $txpver => $endpointData) {
            $newJson['legacy'][$txpver]['version'] = txpspecialchars($endpointData['version']);
            $newJson['legacy'][$txpver]['datePublished'] = txpspecialchars($endpointData['datePublished']);

            if (!empty($endpointData['downloadUrlPhp'])) {
                $newJson['legacy'][$txpver]['endpointUrl'] = txpspecialchars($endpointData['downloadUrlPhp']);
            } elseif (!empty($endPointData['downloadUrlTxt'])) {
                $newJson['legacy'][$txpver]['endpointUrl'] = txpspecialchars($endpointData['downloadUrlTxt']);
            }
        }
    }
}
$out = json_encode($newJson);
file_put_contents('/path/to/new/file.json', $out);

EDIT: there's no defeinsive coding around the 'version' and 'datePublished' so you might need to add a corresponding if(!empty()) around each.

philwareham commented 4 years ago

@bloke the output is now working as expected I think. See https://plugins.textpattern.com/plugins/com_connect/json for an example (it's also got a superseded flag on it just for testing).

philwareham commented 4 years ago

Actually, thinking about the other discussion around endpoints, should we actually favour the TXT files here (i.e. check for them first, then fall back to PHP)? Seems wise.

Bloke commented 4 years ago

Yeah, we could go for .txt first if it's available, then php/zip.

That endpoint is still showing as invalid to me:

{
  "name": "com_connect",

  "supersededBy": {
    "name": "shortcodes",
    "location": "https:\/\/example.com\/example.html"
  },

  "stable": {
    "version": "4.6.0",

    "verifiedMinTxpCompatibility": "4.6",
    "datePublished": "2020-01-31",
    "endpointUrl": "https:\/\/github.com\/textpattern\/com_connect\/archive\/4.6.0.zip"
  }

,{
    "legacy": {
        "4.6": {
            "version": "v4.5.1",
            "datePublished": "2016-12-03",
            "endpointUrl": "https:\/\/github.com\/textpattern\/com_connect\/archive\/v4.5.1.tar.gz"
        },
        "4.5": {
            "version": "v4.5.0.0-beta.4",
            "datePublished": "2016-12-02",
            "endpointUrl": "https:\/\/github.com\/textpattern\/com_connect\/archive\/v4.5.0.0-beta.4.tar.gz"
        }
    }
}
}

Seems there's an extra pair of braces after the comma surrounding the legacy section. That's probably because the entire $json2 has been run through json_encode() which creates an entire json structure, including surrounding braces. What we need to do is either:

1) paste the legacy fields into the existing JSON data; or 2) copy the data we want from the existing JSON data into the start of the new array

and then do a single json_encode() on the result.

philwareham commented 4 years ago

Can't we just trim those braces out - I've spent way too much time on this already? :)

Bloke commented 4 years ago

Sure, try it!

Bloke commented 4 years ago

P.S. if we're sure this content isn't going to be consumed by, say, JavaScript then we could also pass the output through the JSON_UNESCAPED_SLASHES filter for neatness during encoding.

Hopefully json_decode() handles the backslashes for us so we don't have to faff with stripping them out manually when we come to use these files.

EDIT: second thoughts, let's leave them in. Ignore me. We might want to grab this info via Ajax.

philwareham commented 4 years ago

OK, the fixes have been done. Endpoints are now also TXT files where available, falling back to PHP if not.