Roadmap?

[HMAZonderland]

I've been using this XML-RPC library for quite some time now. I'm wondering when are new features expected? What can we expect? What are the plans with this project at all? A heads up would be nice.

[gggeek]

Hi

the lib having been available for such a long time, it is quite stable by now, and quite feature-complete, if you ask me. But of course I am open to requests and suggestions.

If you take a look at http://gggeek.github.io/phpxmlrpc/, there is a "roadmap" section. Tbh most of the ideas in there have been there for years - I did not have time to implement them, and no one clamored to get them implemented.

A completely different take is to revamp this library to bring it to the "modern php" era:

• raise minimum php version to 5.3
• introduce namespaces
• replace .inc files with .php ones
• possibly rebase the http layer on guzzle or other existing libraries
• etc

This is a bit more "controversial", as

• if we do that, api breackage will happen. even if we strive to keep it at a minimum
• there are already other exiting, more modern / slim / fast xmlrpc libraries out there
• xmlrpc is incorporated into every php as a language extension anyway
• rest has kind of superseded xmlrpc

In other words: is that worth doing at all? How many end users would we gain?

[gggeek]

Btw, I am opening separate github issues for all of the features mentioned above

[gggeek]

A lot of work has gone into the 'v4' - available as branch 5.3 so far. Still a lot to be done before I' consider it ready for release.

Here's the snapshot of the "roadmap" (more of a detailed checklist, really) as of today:

bugs

• parsing fails with hhvm for Windmühle... see bug https://github.com/facebook/hhvm/issues/4837
• debugger+lib problem with Windmühle a) ok: let client add proper charset in xml prolog when sending request (actually in http header) b) ok, at least partially: client debug mode 2 fails to echo request payload (also decoded server debug info) => now it will lie on some characters! ok: when us-ascii is used, a warning is generated c) ok: test using proper utf8 characters in the debugger payload form instead of latin-1, eg kosme => the client sends xml entities encoded, plus utf8 charest in prolog d) ok: do the same when debugger is used via POST, not GET e) ok: test methods with latin-1 characters in their name f) also test utf8 characters in the method name (needs the server to work in UTF-8 mode internally) g) ok: usage of htmlentities: the assumed charset changed with newer php versions => at least now we allow it to be set, and tolerate bad chars... h) ok: review creator call for xml parser i) set up tests for all of this: guess_encoding, guess_encoding when mbstring returns false, japanese requests/responses (non-utf8) j) ok: fix known limitation with latin-1 declared in prolog k) test: sending xml with bom: is it transformed to utf8 correctly?
• new xmlrpc\value(-1) = empty value => but how can we fix it without breaking API compat?

refactor

MINOR

• no more 1-letters vars anywhere
• demoes: use only known existing servers
• also: use camelCase names for phpxmlrpc members ?
• finish removal of global vars (server.php, wrapper.php)
• finish cleanup of xmlrpc.inc: server/discuss.php MAJOR
• set internal encoding to utf8 by default (or to default_charset)
• check out some for of dependency injection for eg. charset encoder and logger
• lib: more protected methods and members, overexpose them in subclasses
• also: make new method names camelCase, leave old method names in subclasses
• also: change order/defaults of method arguments where it makes sense
• create Request::AddParams() ?
• create Client::call($method, array()) ?
• request: make debug a private member (add setDebug())
• client: add a setOption method, move all setXXX to child class?
• client: add a mode where exceptions are thrown for http/parsing errors
• value: simplify handling: generic ::size(), support Iterator interface? subclasses of Value?
• make generated functions wrapping calls throw exceptions on receiving errors by default

debugger

• move debugger to utf8 output by default

testing

• silence errors expected during testing for better phpunit output on travis => note that they do not display on localhost, only on travis
• use output buffer and debug mode to dump http payloads when tests fail without the need for the DEBUG flag => mostly done; needs better testing and would be nice if we could handle better curlopt_stderr
• remove dirs /vendor and /tests from coverage reports
• the demo and debuger code does not respect code coverage instructions
• disallow multitest class to execute again single tests (after they have already been carried out by localhosttest)
• allow usage of std way to define parameters for phpunit (besides CLI options)
• check out how to include php-code-coverage-verifier in automated tests
• more tests for debugger/action (execute actual calls)
• add tests for tests/benchmark.php (only the encoding/decoding part)
• create a 'legacyAbuse' test suite whih uses all public-but-should-not-have-been-so stuff
• if running testsuite with a non-bundled phpunit, tests will fail because autoload will not be properly set up -> create phpunit xml config
• check usage of afl for lib testing

travis

• cat hhvm error_log after failed tests (where is it?)

codeclimate, scrutinizer

• remove tests from quality analysis ? (all except /lib and /src?)

build process

• move from makefile to pakefile
• doc: use AsciiDoc as main doc format instead of docbook
• doc: seems like highlight.php does not highlight code anymore (at least in pdf)
• doc: put APIdocs inside the main zipball?
• fix PATH problem to egrep => move to php?
• fix 'make tag' in makefiles to use git

demos

• add client-proxy
• more refactoring of comment.php

functionality

• allow 302s
• look at code from openads (does it support https with plain sockets)
• test and document the fact that incoming charset can be != the 3 basic ones if mb_string is enabled
• support for closures
• test 3.0.1 vs 4 for speed/memory (many calls and huge calls)

misc

• save somewhere todo.txt in main repo

docs

• verify. docs html: does it give warnings in strict mode?
• add an INSTALL chapter to html docs
• update API docs via phpdoc / jsdoc and ulpoad them on the site document:
• xmlrpc server in phpvals mode converts:
  • with auto_dates
  • without null
  • with configurable option
• debug info is not always 100% accurate

sites

• update code on sf.net and gggeek
• web site: add requirements (link to docs), also link to changelog
• js web page: add "features"
• add piwik to website

[gggeek]

Further updates now that version 4 is out:

bugs

• parsing fails with hhvm for Windmühle... see bug https://github.com/facebook/hhvm/issues/4837
• OK: debugger+lib problem with Windmühle a) ok: let client add proper charset in xml prolog when sending request (actually in http header) b) ok, at least partially: client debug mode 2 fails to echo request payload (also decoded server debug info) => now it will lie on some characters! => document it ok: when us-ascii is used, a warning is generated c) ok: test using proper utf8 characters in the debugger payload form instead of latin-1, eg kosme => the client sends xml entities encoded, plus utf8 charset in prolog d) ok: do the same when debugger is used via POST, not GET f) ok: test utf8 characters in the method name g) ok: usage of htmlentities: the assumed charset changed with newer php versions => at least now we allow it to be set, and tolerate bad chars... h) ok: review creator call for xml parser j) ok: fix known limitation with latin-1 declared in prolog
• fix: the same problem applies to the self-documenting server (in the 'extras' package)
• new xmlrpc\value(-1) = empty value => but how can we fix it without breaking API compat?

refactor

MINOR

• no more 1-letters vars anywhere
• use $req instead of $m in tests
• also: use camelCase names for phpxmlrpc members ? MAJOR
• set internal encoding by default to default_charset instead of UTF8?
• check out some form of dependency injection for eg. charset encoder and logger
• lib: more protected methods and esp. members, overexpose them in subclasses => start by declaring direct access deprecated and adding getters/setters
• request: make debug a protected member (since we added setDebug())
• also: make method names of new classes camelCase, leave old method names in subclasses => eg. add_to_map, xmlrpc_debugmsg, error_occurred and xml_header in Server
• move arraysize, structsize, structeach, arraymem, structmem, structmemexists to subclass? => in v5?
• change order/defaults of method arguments where it makes sense (such as: ...)

new functionality

• date helper: test what happens when a date object is passed in instead of a timestamp
• create Request::AddParams() ?
• create Client::call($method, array()) ?
• client: add a setOption method, move all setXXX to child class ? => useful for later creating an sf Bundle, as well a a generic 'webservice' parent classes layer
• client: add a mode where exceptions are thrown for http/parsing errors (as well as for error responses?)
• Value: add an 'auto' mode where the valtyp is guessed
• improve wrapping of functions:
  • check: can we build a wrapped function even without its introspection? => e.g. do not produce a signature
  • check: how can we generate signatures for wrapped php funcs which use varargs + phpdoc?
  • when generating source code for a wrapped function, add phpdoc as well
  • wrapped php functions: add an option to delegate to the server validation of number of params
  • when we wrap functions which are already xmlrpc-aware, output is fine but function input does get fed to the function as plain php val => we could avoid decoding it (or at least print a warning)
  • allow wrapping of closures when returning source code via f.e. https://github.com/jeremeamia/super_closure
• improve wrapping of methods:
  • allow generated functions wrapping method calls to throw exceptions on receiving errors (new option)
  • allow wrapping of methods which have no sigs (using varargs?) - new option
• client: add default timeout, protocol to be used for send (if they do not exist already)
• client: move to streams usage instead of fsockopen (see ezcontentstaging) to allow different transports:
  • finish usage of http10 for https: support client's https options
  • use fopen instead of stream_socket_client when protocol is 'file'
  • add a slowloris test: check how current code deals with timeouts of the server => shall we check using stream_get_info ?
• value: support arrayAccess interface => add tests and docs; clean up code and finalize decisions
• value: use subclasses of Value? => make it easier to create values of desired type
• allow 302s
• look at code from openads (does it support https with plain sockets)
• test 3.0.1 vs 4 for speed/memory (many calls and huge calls)

backwards compat

• can not set any more server system.xx dispatch map or system.getCapablities (was doable via global vars)

debugger

testing

• test methods with latin-1 characters in their name (needs the server to work in latin-1 mode internally)
• set up tests for:
  • guess_encoding when mbstring returns false
  • ok: guess_encoding returning iso-8859-* => the php manual explains it confusedly. it is detected as 8859-1 or 8859-15, not as 8859-*
  • guess_encoding with xml with bom
• add tests for arrayiterator access (also for phpxmlrpcval style) => ok ?
• add tests for arrayaccess interface
• silence errors expected during testing for better phpunit output on travis => note that they do not display on localhost, only on travis
• use output buffer and debug mode to dump http payloads when tests fail without the need for the DEBUG flag => mostly done; needs better testing and would be nice if we could handle better curlopt_stderr
• remove dirs /vendor and /tests from coverage reports
• the demo and debugger code does not generate code coverage info. move to pre- and post- includes?
• disallow multitest class to execute again single tests (after they have already been carried out by localhosttest)
• allow usage of std way to define parameters for phpunit (besides CLI options)
• check out how to include php-code-coverage-verifier in automated tests
• more tests for debugger/action (execute actual calls)
• more tests for server/discuss (execute actual calls)
• add tests for tests/benchmark.php (only the encoding/decoding part)
• create a 'legacyAbuse' test suite which uses all public-but-should-not-have-been-so stuff
• check usage of afl for lib testing
• test running testsuite on phpunit 3.7 ?
• increase code coverage: iso8601decode, exportGlobals, Encoder::encode, Server::...

travis

• cat hhvm error_log after failed tests (where is it?)

codeclimate, scrutinizer

• remove tests from quality analysis ? (all except /lib and /src and /demo and /debugger?)

build process

demos

• demo more features: charset manipulation encoding/decoding: base64 data and datetimes basic auth https request compression response::parseresponsefile cookies

misc

• save somewhere todo.txt in main repo
• test speed of wrapped functions vs manual ones (both server side and client side)

docs

format:

• use AsciiDoc as main doc format instead of docbook:
  • check how the generated pdf compares to the original one
  • no TOC visible on github: is it normal or an effect of it being set on the left?
• is source code in pdf highlighted? (it is in hml, but can use different highlighters)
• verify. docs html: does it give warnings in strict mode?

content:

• new cotent structure:
  • creating xmlrpc values => different ways, date helpers
  • sending requests => different types of responses returned
  • dealing with character sets
  • advanced http features
  • debugging
  • using the debugger
  • creating servers => dispath map, what to return
  • creating proxy objects
  • reserverd methods => move to appendix
  • files in the distribution => move to appendix ?
  • global variables => move to appendix
  • new appendix: removed methods
  • to remove: chapter 11 (keep a title... ?); appendix C (move to val creation); move appendix D to chapter;
• say: xmlrpc server in phpvals mode converts:
  • with auto_dates
  • without null
  • with configurable option
• say: debug info is not always 100% accurate
• document better all charset stuff, including the fact that incoming charset can be != the 3 basic ones if mb_string is enabled (and how to tell it to the server)
• documet all the ways a php function/method can be registered into the server
• document BC of wrap_php_class and its new param; new methods in value class: count(), breackage of foreach; other?
• document: wrapping native php functions => false, resources => false, closures bound to objects => same as plain closures (ok with non-return-source)

sites

• js web page: add "features"
• add piwik to website ?

[gggeek]

...and here is one more update:

P0

• remove/update both code and downloads on sf.net
  • add readmes/redirects for both web pages and downloads

P1

• rename master branch

• modernize the extras package

  • split it out in different packages? If not, shuffle around 'library' code and demo code
  • base it on phpxmlrpc 4.x api
  • introduce 'BC API Compat layer'
  • add unit tests (esp. for epi compat layer)

P2

• test pake build actions: does everything still work ?

• add a github hook that checks version nr before release

  • also: a ci pipeline for generating the docs
  • also: can we use automated tools to scan for php 8.0 compatibility problems? see a list here: https://thevaluable.dev/code-quality-check-tools-php/

• improve test container build:

  • make 'start' wait until composer is done running (via entrypoint.sh, as in kezmb)
  • set a nice hostname in the container
  • make all setup shell scripts idempotent
  • check use of docker scan for security
  • q: do we really need to make 'test' user a sudoer? (a: it is as in travis...)

P3

general

• socket timeout: do we need to use stream_getinfo to properly detect it? (test & add test for it)

• add tests for cookie names with spaces => add fix in 431 (??)

• test creating client using full url 'http11://etc'. If it works, document it

• test what is returned by multicall() when return type is xml

• date helper: test what happens when a date object is passed in instead of a timestamp

• emit a warning when encoding a resource to json in api emulation mode; try to error out when encoding invalid utf8

• allow an easy way to disable demo/server/xx.php (eg. proxy) for installing on places such as gggeek.altervista.org ? (its as easy as deleting files from disk...)

bugs

• problem with Windmühle in the self-documenting server (in the 'extras' package)

• backport fix for spaces in cookie value in client->setCookie to 3.1 ?

• WONTFIX: new xmlrpc\value(-1) = empty value => but how can we fix it without breaking API compat?

• WONTFIX: parsing fails with hhvm for Windmühle... see bug https://github.com/facebook/hhvm/issues/4837

backwards compat

• can not set any more server system.xx dispatch map or system.getCapablities (was doable via global vars)

new functionality

NB: all big refactoring and new functionality is moved off to V5 implementation. Possibly left for V4:

• add support for setting ssl version in socket mode
• add support for 'options' param in json_encode in api emulation mode
• allow 302s
• look at code from openads (it most likely support https with plain sockets)

perfs

• test 3.1.0 vs 4 for speed/memory (many calls and huge calls)
• test speed of wrapped functions vs manual ones (both server side and client side)

testing

• clean up test code syntax: use modern phpunit classes and methods

  • replace _fail and _run with TestListener Polyfill

• install 'de' locale for the one skipped test in Docker

• add more tests for mixed http features: basic+digest auth on https/socket, helpers (date enc/decoding) etc...

• add test case for client setting of curl extra options

• add test cases for json_last_error

• add test case for phpvals validation of types server-side in both jsonrpc and xmlrpc (eg: empty structs)

• add test case for mixed-type servers in both jsonrpc and xmlrpc

• add gui test for uri-too-long-error

• test methods with latin-1 characters in their name (needs the server to work in latin-1 mode internally)

• set up tests for:

  • guess_encoding when mbstring returns false
  • ok: guess_encoding returning iso-8859- => the php manual explains it confusedly. it is detected as 8859-1 or 8859-15, not as 8859-
  • guess_encoding with xml with bom

• add tests for arrayiterator access (also for phpxmlrpcval style) => ok ?

• add tests for arrayaccess interface

• use output buffer and debug mode to dump http payloads when tests fail without the need for the DEBUG flag => mostly done; needs better testing and would be nice if we could handle better curlopt_stderr

• disallow multitest class to execute again single tests (after they have already been carried out by localhosttest)

• allow usage of "std way" to define parameters for phpunit (besides env vars) -> rename phpunit.xml to .dist.xml

• check out how to include php-code-coverage-verifier in automated tests

• more tests for debugger/action (execute actual calls)

• more tests for server/discuss and server/proxy (execute actual calls)

• create a 'legacyAbuse' test suite which uses all public-but-should-not-have-been-so stuff

• check usage of afl for lib testing

• increase code coverage: iso8601decode, exportGlobals, Encoder::encode, Server::...

• allow to run tests on php 5.3

  • check version of apache installed with precise, trusty: we most likely need a different config
  • esp. mod_proxy_fcgi
  • same for php-fpm
  • also: need a separate composer.json to bring in phpunit 4?
  • also: yoast/phpunit-polyfills only goes as far basck as php 5.5

travis

• check where php_fpm error logs are and diplay them (so far they have not been found where expected...)
• do we need/want to run phpunit with --stderr ?

codeclimate, scrutinizer, symfonyinsight & friends

• remove tests from quality analysis ? (all except /lib and /src and /demo and /debugger?)
• re-enable codeclimate on Travis after it becomes compat with php 8.0

debugger

• rebuild GUI based on a Bootstrap Admin theme ?

demos

• add to vardemo: date objects, auto-conversion, anon obj to struct
• demo/server/discuss.php: replace dba with sqlite
• demo more features: charset manipulation encoding/decoding: base64 data and datetimes basic auth https request compression response::parseresponsefile cookies

docs

format:

• use AsciiDoc as main doc format instead of docbook:
  • check how the generated pdf compares to the original one
  • no TOC visible on github: is it normal or an effect of it being set on the left?
• is source code in pdf highlighted? (it is in hml, but can use different highlighters)
• verify. docs html: does it give warnings in strict mode?

content:

• new cotent structure:
  • creating xmlrpc values => different ways, date helpers
  • sending requests => different types of responses returned
  • dealing with character sets
  • advanced http features
  • debugging
  • using the debugger
  • creating servers => dispath map, what to return
  • creating proxy objects
  • reserverd methods => move to appendix
  • files in the distribution => move to appendix ?
  • global variables => move to appendix
  • new appendix: removed methods
  • to remove: chapter 11 (keep a title... ?); appendix C (move to val creation); move appendix D to chapter;
• say: xmlrpc server in phpvals mode converts:
  • with auto_dates
  • without null
  • with configurable option
• say: debug info is not always 100% accurate
• document better all charset stuff, including the fact that incoming charset can be != the 3 basic ones if mb_string is enabled (and how to tell it to the server)
• documet all the ways a php function/method can be registered into the server
• document BC of wrap_php_class and its new param; new methods in value class: count(), breackage of foreach; other?
• document: wrapping native php functions => false, resources => false, closures bound to objects => same as plain closures (ok with non-return-source)

sites

• make design responsive
• js web page: add "features"
• add piwik to website ?

misc

• save somewhere todo.txt in main repo ?

[gggeek]

One more update:

P1

• CD script does not connect to alt.org -> test on next release?

• test/fix what is returned by multicall() when return type is xml

  • also: add to multicall responses the missing httpResp. data

• fix handling of parameters_types in the dispatch map

• test: use mbstring (or iconv) if available to support more in/out charsets (eg. internal cp1252, ext. utf8) see code in https://github.com/gggeek/phpxmlrpc/issues/42 - it might already be working!

  • add it to demos

• (same as above?) make Charset helper support CP1252 and the full range of chars including C1 (either here on in polyfill-xmlrpc)

  • move to loading of charset tables from external files ?
  • simplify the isset() test in the jit loading of Charset tables

• date helper: test what happens when a date object or string is passed in instead of a timestamp. Is it useful to support that?

• check out support of fsockopen-mode for http/2 and all other client features such as support for redirects

• make debugger smarter about presence of js files (and jsonrpc files) ?

  • need first a 2nd release of the extras package with ajax support reimplemented (really?)
  • while at it, why not move the debugger to a separate package altogether ? (done in v5...)

• make wrapper be able to generate code with different namespace (for jsonrpc debugger)

  • check as well all other methods which create Value/req/Resp: are they used by jsonrpc subclasses? if so, do they work?

• create a custom server: both self-documenting and jsonrpc+xmlrpc -> a "combo" server... (either as 'demo' in the extras package, as part of the debugger container, or as a standalone container)

  • install jsxmlrpc via npm or taskfile, or is it enough to load it from cdn ?
  • install all packages in proper dirs - ie. outside web root ? (without breaking the debugger if included)
  • deploy it to gggeek.altervista.org instead of demo/server/server.php ?

• tests:

  • add a test following a 301
  • add tests for Response::httpResponse and friends
  • add test for encoding of chars 0-31 in utf8->ascii mode
  • also: do encode character 127
  • add tests for v3 api fixes of release 4.5.1
  • add test for catching Error
  • add test for method handler throwing exception with error 0
  • add tests that actually make all server demos execute stuff (eg. discuss.php, codegen.php)

• allow to avoid population of httpResponse in both Request and Response in non-debug scenarios

• add Client::setTimeout method

• allow Client to use specific decoding options when return_type = XMLParser::RETURN_PHP (esp. for date decoding)

• replace usages of extension_loaded with function_exists, to allow polyfills

• make all setXXX methods return $this

• allow Server::add_to_map to set parameters_types

• make all setCharsetEncoder methods static (why was this not done since day one?)

• add setLogger() to classes missing it (HTTP, XMLParser); mention it in manual / add a demo for it

• fix Client sometimes logging an error message but then continuing instead of returning a Fault, in sendPayloadSocket, prepareCurlHandle

• add options to wrapXmlrpcServer to have the generated methods throw on fault. Also: make the generated code use phpvals by default. Also, when generating code wrapping an object instance for codegen usecase, allow a simpler way to set the obj on use

• add demo servers for symfony and laravel

• user manual:

  • add a "BC promise" section

• api docs: improve HP after generating it with phpdoc: in main column add version nr, remove "packages"

• add taskfile action to set up github hook

• add gha workflow to trigger release of a new version of the debugger container upon a release of the main lib

• make sure tests/index.php can not be triggered by non-phpunit requests

• make Request::$httpResponse private and add protected __get (if possible) to log a deprecation warning

• deprecate usage of params $timeout and $method in send(), log a deprecation warning when used

• check if we can add warnings upon use of other deprecated methods and properties

• add setOption to both Client and Server, deprecate all access to setXXX methods and $xxx members (use call, get and co.)

• release 4.10

P2

• use userforks.io to get list of downstreams from gh: composer users, forks, code search for usages of bad methods

• add a 'tight/fast' serialization mode as static var in PXR: serialize strings without , ints as i4

• add constants for all string-valued parameters (such as return types)

• bump requirements to 5.4, introduce traits such as loggerAware

• test all usage of is_a; replace with instanceof ?

• review cookie handling code against the latest cookies rfc (6265) - drop support for dead cookies specs

• add to verify_compat.php testing for recent lib features, e.g. presence of specific curl constants

• make its output nicer when run from the cli

• check if we can reduce usages of preg_match - in a single call script, they take up 14% of wall time

• run static code analyzers:

  • snyk: OK
  • sonarqube: OK
  • others: ...

• check speed diff vs. lstrojny/fxmlrpc

  • also: the parser written a while ago by bdunogier (where is it?)

• add curl multiplexing when using h2 in the sendParallel demo

• change the files which do direct loading of vendor/autoload.php to allow for 'installed as dep' config

  • also: what if vendors dir is named something else? check composer docs about that

• improve test container build:

  • make 'start' wait until composer is done running (via entrypoint.sh, as in kezmb)
  • always remove composer.lock, in case the php env has changed ?
  • set a nice hostname in the container
  • make all setup shell scripts idempotent
  • check use of docker scan for security

• figure out y stream_context tests fail with php 5.4->7.1 on focal and later

• rename master branch to main ?

• should we default to http2 instead of h2 ?

• move sendParallel to extras, or to main Client ?

• allow Client to throw in case of errors or Fault responses ? (demo file proxy.php might be sufficient...)

  • add this to demo code
  • also add a "call" method to simplify transparent usage ?

P3

general

• socket timeout: do we need to use stream_getinfo to properly detect it? (test & add test for it)
• add tests for cookie names with spaces => add fix in 431 (??)
• allow autoloading of v3 classes via composer (using 'classmap') ? NB: autoloading them has an impact on global vars!
• add taskfile commands to
  • start the php webserver in the debugger dir (or top-level dir?) - needs to many optional params: dir, port, ip...

bugs

• backport fix for spaces in cookie value in client->setCookie to 3.1 ?

• WONTFIX: new xmlrpc\value(-1) = empty value => but how can we fix it without breaking API compat?

• WONTFIX: parsing fails with hhvm for Windm�hle... see bug https://github.com/facebook/hhvm/issues/4837

new functionality & refactoring

NB: all big refactoring and new functionality is moved off to V5 implementation. Possibly left for V4:

• move Encoder::decodeXml to XMLParser ?
  • also: add a nicer API to XMLParser, which does not leak $_xh
• move Request::ParseResponse to the Client
• add support for setting ssl version in socket mode
• look at code from openads (it most likely support https with plain sockets)
• support brotli compression if the relevant function exists (see https://github.com/kjdev/php-ext-brotli)

perfs

• test speed of wrapped functions vs manual ones (both server side and client side)
• see item above about comparison w. different parsers
• can we parse a stream incrementally instead of using a string?
• test 3.1.0 vs. 4 for speed/memory (many calls and huge calls)

backwards compat

• can not set any more server system.xx dispatch map or system.getCapabilities (was doable via global vars)

testing

• install 'de' locale for the one test skipped in Docker (also, in GHA)

• clean up test code:

  • use modern phpunit classes and methods
  • replace _fail and _run with TestListener Polyfill if possible
  • use utf8 as default internalencoding, and reset it on every test via setup (in case one test fails, avoid cascades)
  • clean up testCase classes: merge the class that does http calls with the one that does xmlrpc calls?
  • reduce number of env params: HTTPSSERVER is useless now, etc...
  • move testing of v3 api to its own separate test class
  • remove usage of output buffering?
  • use output buffer and debug mode to dump http payloads when tests fail without the need for the DEBUG flag => mostly done; needs better testing and would be nice if we could handle better curlopt_stderr
  • disallow multitest class to execute again single tests (after they have already been carried out by localhosttest)
  • allow usage of "std way" to define parameters for phpunit (besides env vars)

• add tests for dispatch map with a registered callable with a specific calling convention

• add more tests for mixed http features: basic+digest auth on https/socket, helpers (date enc/decoding) etc...

• add test case for client setting of curl extra options

• test more server features (eg. is there any test of return_phpvals calling convention?) eg. function which adds user-defined debug

• add test case for phpvals validation of types server-side in both jsonrpc and xmlrpc (eg: empty structs)

• add test case for mixed-type servers in both jsonrpc and xmlrpc

• add gui test for uri-too-long-error

• test methods with latin-1 characters in their name (needs the server to work in latin-1 mode internally)

• set up tests for:

  • guess_encoding when mbstring returns false
  • ok: guess_encoding returning iso-8859- => the php manual explains it confusedly. it is detected as 8859-1 or 8859-15, not as 8859-
  • guess_encoding with xml with bom

• add tests for arrayiterator access (also for phpxmlrpcval style) => ok ?

• add tests for arrayaccess interface

• more tests for debugger/action (execute actual calls)

• more tests for server/discuss and server/proxy (execute actual calls)

• increase code coverage: iso8601decode, exportGlobals, Encoder::encode, Server::...

• create a 'legacyAbuse' test suite which uses all public-but-should-not-have-been-so stuff

• check out how to include php-code-coverage-verifier in automated tests

• check usage of afl for lib testing

• allow to run tests on php 5.3

  • check version of apache installed with precise, trusty: we most likely need a different config
  • esp. mod_proxy_fcgi
  • same for php-fpm
  • also: need a separate composer.json to bring in phpunit 4 ?
  • also: yoast/phpunit-polyfills only goes as far back as php 5.5 (? we already run tests on 5.4...)

test container

• rename 'docker' user to something else ? check the default username and workspace folder in gha

GHA

• change composer install options ? eg. no-interaction, optimize-autoloads
• do we need/want to run phpunit with --stderr ?
• CD pipeline:
  • OK? test in next release: push as well code to gggeek.altervista.org
  • check: are the release assets tarballs the ones installed via composer ?
  • allow to trigger manual regen via a specific git commit msg or tag syntax? (it is enough to add a space in code...) useful esp. if we separate te job, eg. trigger just updating of code on alt.org
  • suspend release job until tests are finished ?
  • can we create a dedicated user on alt.org, with less write permissions ?
  • remove excess newlines from gh release notes (the ones we use in the md file for staying within 120 chars)

debugger

• rebuild GUI based on a Bootstrap Admin theme ?

• add tooltips with explanation/examples for all fields

• do display somewhere the lib version (both phpxmlrpc and jsonrpc...)

• externalize the debugger to its own composer package ? (it is done in v5...)

demos

• srv: make it more evident how to make server work with all the different ways of registration of php code as method handlers - transform functions.php to register every function in a different way ?
• client: improve which.php: atm it has no real need for either xml handling or http redirects...
• demo more features: character set conversion (atm it is only in vardemo.php) roundtripping of php classes (note: it is in wrapper.php) basic auth https cookies response::parseresponsefile

docs

• test alternatives to phpdoc: https://github.com/ApiGen/ApiGen, https://github.com/theseer/phpdox/

• phpdoc docs: can we avoid generation of "xmlrpc" (or whatever it is named) package ? Use Readme for its homepage ? Also: add lib version nr. and doc generation date as page footer or in hp. Try with a bit of css magic...

• content:

  • reserved methods, dropped functions, error codes => move to appendix ?
  • add a section about security
  • document better all charset stuff, including the fact that incoming charset can be != the 3 basic ones if mb_string is enabled (and how to tell it to the server)
  • document BC of wrap_php_class and its new param; new methods in Value class: count(), breakage of foreach; other? (is that v4 vs v3?)
  • document: wrapping native php functions => false, resources => false, closures bound to objects => same as plain closures (ok with non-return-source)
  • review usage of terms-lists

sites

• make design responsive
• jsxmlrpc web page: add "features"
• add matomo to ggg.alt.org website ?

codeclimate, scrutinizer & friends

• remove tests from quality analysis ? (all except /lib and /src and /demo and /debugger?)
• check out other services for code QA
• move scrutinizer config to inside the project
• add to README the symfonyinsight badge (? it seems not to be free anymore for oss projects) https://insight.symfony.com/projects/c334a2c0-7aa1-49df-98c3-c99657cf44a6
  • also: enable post-commit-hook to run it on each commit
• re-enable codeclimate on GHA after it becomes compat with php 8.1 ? (it seems to be dying...)
  • send PR for that

misc

[gggeek]

Release 4.10.0 has been published today, with a long list of fixes, improvements and deprecations, some of which were listed in the last "roadmap" update above, but definitely not all.

This proves once more that I am comically bad at planning and roadmapping 😅

Speaking in broad terms, this is what I expect to focus on, going forward:

• checking the possibility of having the server component generate automatically the modern-day equivalent of WSDL descriptions (such as Swagger), starting from the method dispatch map. More in general, explore the topic of IDLs and code-generation applied to XML-RPC
• improving the jsonrpc spin-off library, esp. making it compliant with the json-rpc 2.0 spec
• one ore more point releases, to fix bugs and implement feature requests
• one major release (v5), which should drop support for php 5 and the v3-api-compatibility layer
• merge all the changes in the "NG" branch, and release it as stable with a compatibility layer for v5.x

Below here follow the latest update to the "detailed to do list" which I have been posting here for almost eight years!

--

P1 - plan for the next (minor) version

nada!

P2 - wishlist

• add gha workflow to trigger release of a new version of the debugger container upon a release of the main lib

  • same for extras and jsonrpc libs

• deploy to gggeek.altervista.org the demo server from the debugger container next to demo/server/server.php

  • have it automatically uploaded on every tag push to the container's github
  • mention the new url in the docs (pxr, pxr-jsonrpc, pxr-extras, pxr-debugger)
  • is there any value in having it use the ajax-wrapper capabilities ?

• add constants for all string-valued parameters (such as return types)

• tighten parsing of xml-rpc: faulty fault responses, missing methodname, ... (see todos in xmlparser)

  • add tests for those cases

• restore non-deprecated accessor to: Value::$_php_class, Request::$debug ?

• add utility methods: PhpXmlRpc::setParser, setCharsetEncoder (take care about exp/imp globals) ?

• fix XMLParser sometimes logging an error message but then continuing instead of throwing, when setting unsupported options ?

• test all usage of is_a; replace with instanceof ?

• create some demos (client, server) showcasing the new parsing options - esp. using dates-as-objects and strict parsing

• add demo clients, servers for laravel

  • then answer to https://stackoverflow.com/questions/73744099/xmlrpc-library-for-api-in-laravel

• implement all missing http features in client socket mode, such as support for redirects, digest/ntlm auth, etc...

• make Charset helper support CP1252 natively (without mbstring):

  • split the support for CP1252 in a subclass
  • allow it to also load charset tables from external files ?
  • make it easy to decide which class to use by default (e.g. use a PhpXmlRpc static var as class name, or add a static class var in all classes, or do the same as setLogger)
  • make it encode the full range of chars including C1 control chars (either here on in polyfill-xmlrpc)

• allow Client/Server::setOption('parsing_options') to add options for the XMLParser, so that Parser::parse() can be made independent of global state (PhpXmlRpc static variables)

  • do the same for Parser::guessEncoding
  • and for Encoder::decodeXml what about adding them as static Parser vars instead of parse() options? It has a slight speed edge on access...

• remove all the recently added static vars from PhpXmlRpc ?

  • add consts for their default value
  • allow them to be overtaken via method calls (either only in Client/Server/Encoder, or via static method PhpXmlRpc::setOption, but then we'll also need a getOption) - nb: php misses __setStatic
  • what about options which affect serialization: should we introduce a serializer class ? And options used in other classes ?

• add to verify_compat.php testing for recent lib features, e.g. presence of specific curl constants

• make its output nicer when run from the cli

• run static code analyzers:

  • snyk: OK
  • sonarqube: OK
  • qodana: OK
  • others: ...

• run an encoding/decoding pass with a profiler and check where time is spent

  • check if we can reduce usages of preg_match - in a single call script, they take up 14% of wall time

• check speed diff vs. lstrojny/fxmlrpc

  • also: the parser written a while ago by bdunogier (where is it?)
  • also: latest parser vs. v2, v3 parsers
  • both ram and speed (tested on 1/2/2023)
  • plus, can we automate that? (run it on gha, save in gh site branch a graph or textual results)

• check any nonCamelCase method, add an alias and deprecate old name: resp::xml_header, srv:...

• add curl multiplexing when using h2 in the sendParallel demo

• move sendParallel to extras, or to main Client ?

• change the files which do direct loading of vendor/autoload.php to allow for 'installed as dep' config

  • also: what if vendors dir is named something else? check composer docs about that

• improve test container build:

  • make 'start' wait until composer is done running (via entrypoint.sh, as in kezmb)
  • always remove composer.lock, in case the php env has changed ?
  • set a nice hostname in the container
  • make all setup shell scripts idempotent
  • check use of docker scan for security

• figure out y stream_context tests fail with php 5.4->7.1 on focal and later

• rename master branch to main ?

• should we default to http2 instead of h2 ?

v5 - mostly refactoring and cleanups

• bump minimum requirements to php 7.x

  • introduce finally clauses where appropriate

• drop all BC stuff, including the .inc layer

  • fix tests

• move Value::_php_class to a dedicated subclass (use existing git branch which has that already)

• add interfaces for Logger, Parser, Encoder

• remove setting $resp->raw_data on the server-side

• allow Client to throw in case of errors or Fault responses ? (demo file proxy.php might be sufficient...)

  • add this to demo code
  • also add a "call" method to simplify transparent usage ?

• allow the XMLParser to work directly on streams, alter request::parseResponseFile() to follow suit; add demo for large-response parsing (and measure memory difference)

• move Encoder::decodeXml to XMLParser ?

• move Request::ParseResponse, ParseResponseFile to the Client

• add a 'compact/fast' serialization mode as static var in PXR: serialize strings without , ints as i4

• see all other issues from https://github.com/gggeek/phpxmlrpc/issues/73

P3 - low-priority

bugs

• backport fix for spaces in cookie value in client->setCookie to 3.1 ?

• WONTFIX: new xmlrpc\value(-1) = empty value => but how can we fix it without breaking API compat?

• WONTFIX: parsing fails with hhvm for Windm�hle... see bug https://github.com/facebook/hhvm/issues/4837

new functionality & refactoring

NB: all big refactoring and new functionality is moved off to the NG implementation. Possibly left for v4/v5:

• support brotli compression if the relevant function exists (see https://github.com/kjdev/php-ext-brotli)
• look at code from openads (it most likely support https with plain sockets)
• would it make sense to allow the Client to use one internal-charset for request data and a different one for response data? (same for the server)
• allow usage of iconv instead of mbstring everywhere we use mb_convertencoding ? including demo code (tests are more stable with mb than with iconv, as iconv is system-dependent...)
• allow to easily swap out the default helper classes created by the lib: add static $helperClassName in both phpxmlrpc and subclasses ? (it's not like changing a static var is shorter than calling a static method...)
• move support for NIL and other xml-rpc extensions to the Interop class ?
• rename 'xmlrpc_reject_invalid_values' to 'strict_mode' ?

general

• socket timeout: do we need to use stream_getinfo to properly detect it? (check & add test for it. See gh issue)
• allow autoloading of v3 classes via composer (using 'classmap') ? NB: autoloading them has an impact on global vars!
• add taskfile commands to
  • start the php webserver in the debugger dir (or top-level dir?) - needs to many optional params: dir, port, ip...

perfs

• test speed of wrapped functions vs manual ones (both server side and client side)
• see item above about comparison w. different parsers
• in wrapXmlrpcServer, make the generated code use phpvals by default instead of going through the encoder

backwards compat

• can not set any more server system.xx dispatch map or system.getCapabilities (was doable via global vars)

testing

• clean up test code:

  • use modern phpunit classes and methods
  • replace _fail and _run with TestListener Polyfill if possible
  • use utf8 as default internalencoding, and reset it on every test via setup (in case one test fails, avoid cascades)
  • same of every other test which sets a "global-status" variable
  • clean up testCase classes: merge the class that does http calls with the one that does xmlrpc calls?
  • reduce number of env params: HTTPSSERVER is useless now, etc...
  • move testing of v3 api to its own separate test class
  • same for testing NIL and Wrapper
  • use output buffer and debug mode to dump http payloads when tests fail without the need for the DEBUG flag => mostly done; needs better testing and would be nice if we could handle better curlopt_stderr
  • disallow multitest class to execute again single tests (after they have already been carried out by localhosttest)
  • allow usage of "std way" to define parameters for phpunit (besides env vars)

• investigate: when calling wrap_xmlrpc_method and wrapping fails, phpunit hangs (but outside of phpunit all is ok)

• add test: validation of invalid methodname

• test methods with latin-1 characters in their name (needs the server to work in latin-1 mode internally) ? note that utf8 and latin1 chars are not allowed by default in method names...

• add test: response with not-closed server-debug comment

• check: any Server tests which have been made redundant by more extensive parsing tests?

• add test for encoding of chars 0-31 in utf8->ascii mode

• also: do encode character 127

• add tests for v3 api fixes of release 4.5.1 and all fixes up to 4.9.5

• add tests for dispatch map with a registered callable with a specific calling convention => ok?

  • check case of default dispatch map = phpvals, and a single function using xmlrpcvals

• add more tests for mixed http features: basic+digest auth on https/socket, etc...

• test more server features, eg. function which adds user-defined debug

• add test case for phpvals validation of types server-side in both jsonrpc and xmlrpc (eg: empty structs)

• add test case for mixed-type servers in jsonrpc (here ??)

• add gui test for uri-too-long-error

• set up tests for:

  • guess_encoding when mbstring returns false
  • ok: guess_encoding returning iso-8859- => the php manual explains it confusedly. it is detected as 8859-1 or 8859-15, not as 8859-
  • guess_encoding with xml with bom

• encoder tests: add testing of encoding/decoding with options, eg. auto_dates

• add a test for interop error codes

• add tests for arrayiterator access (also for phpxmlrpcval style) => ok ?

• add tests for arrayaccess interface => ok ?

• more tests for debugger/action (execute actual calls)

• more tests for server/proxy (execute actual calls)

• increase code coverage: iso8601decode, exportGlobals, Encoder::encode, Server::... (review coverage reports 1st)

• create a 'legacyAbuse' test suite which uses all public-but-should-not-have-been-so stuff

• check out how to include php-code-coverage-verifier in automated tests

• check usage of afl for lib testing

• verify: can tests still run on ubuntu precise ? (is stock php version >= 5.4 ?)

  • check version of apache installed with precise, trusty: we most likely need a different config
  • esp. mod_proxy_fcgi
  • same for php-fpm
  • also: yoast/phpunit-polyfills only goes as far back as php 5.5 (? we already run tests on 5.4...)

• create a self-contained suite of invalid xml-rpc requests/responses, which can be used for parser validation, eg. values not valid according to the spec, bad element nesting, unknown elements, etc (leaving outside invalid xml and charset encoding issues, or at least storing them separately)

test container

• rename 'docker' user to something else ? check the default username and workspace folder in gha

GHA

• change composer install options ? eg. no-interaction, optimize-autoloads
• do we need/want to run phpunit with --stderr ?
• CD pipeline:
  • check: are the release assets tarballs the ones installed via composer ?
  • allow to trigger manual regen via a specific git commit msg or tag syntax? (it is enough to add a space in code...) useful esp. if we separate the job, eg. trigger just updating of code on alt.org
  • suspend release job until tests are finished ?
  • remove excess newlines from gh release notes (the ones we use in the md file for staying within 120 chars)

debugger

• rebuild GUI based on a Bootstrap Admin theme ?

• add tooltips with explanation/examples for all fields

• externalize the debugger to its own composer package ? (it is done in NG...)

demos

• client: improve which.php: atm it has no real need for either xml handling or http redirects...
• demo more features: character set conversion (atm it is only in vardemo.php) roundtripping of php classes (note: it is in wrapper.php) basic auth https cookies response::parseresponsefile
• add reverse-proxy server demo (same code as in extras ;-)) ?
• add tests which set up Sf and run the Sf demo code

docs

• test alternatives to phpdoc: https://github.com/ApiGen/ApiGen, https://github.com/theseer/phpdox/

• phpdoc docs: can we avoid generation of "xmlrpc" (or whatever it is named) package ? Use Readme for its homepage ? Also: add lib version nr. and doc generation date as page footer or in hp. Try with a bit of css magic...

• content:

  • complete the list of server opts, static vars in PhpXmlRpc
  • reserved methods, dropped functions, error codes => move to appendix ?
  • add a section about security. including: limiting curl access to local files (test it again w. 'normal' srv string)
  • document better all charset stuff, including the fact that incoming charset can be != the 3 basic ones if mb_string is enabled (and how to tell it to the server), mention global settings to tweak for guessing incoming charset
  • document BC of wrap_php_class and its new param; new methods in Value class: count(), breakage of foreach; other? (is that v4 vs v3?)
  • document: wrapping native php functions => false, resources => false, closures bound to objects => same as plain closures (ok with non-return-source)
  • review usage of asciidoc terms-lists
  • document better usage of arrayaccess for scalar values ?

sites

• make design responsive
• jsxmlrpc web page: add "features"
• add matomo to ggg.alt.org website ?

codeclimate, scrutinizer & friends

• remove tests from quality analysis ? (all except /lib and /src and /demo and /debugger?)
• check out other services for code QA
• move scrutinizer config to inside the project ?
• add to README the symfonyinsight badge (? it seems not to be free anymore for oss projects) https://insight.symfony.com/projects/c334a2c0-7aa1-49df-98c3-c99657cf44a6
  • also: enable post-commit-hook to run it on each commit
• re-enable codeclimate on GHA after it becomes compat with php 8.1 ? (it seems to be dying...)
  • send PR for that