PQRS Importer

[samlikins]

Finalize ClaimsToOEMR repository code and integrate into codebase.

[samlikins]

Added Issue #2: Update Raw to integrate Interchange Data specifications

[leebc]

@samlikins : Do you have an estimated timeline on this yet?

[samlikins]

Closed Issue #2

[samlikins]

Added Issue #5: Refactor structure of classes to simplify abstraction/processing

[samlikins]

Closed Issue #5

[samlikins]

Added Issue #8: Review and modify Cache class

[samlikins]

Closed Issue #8 to begin integrating ClaimsToOEMR project, further corrections will continue.

[samlikins]

Added composer components and integrate autoloader into globals.

[samlikins]

Testing environment setup and new PQRS Gateway instance installed for integration; composer integration successful.

[aethelwulffe]

Means that it pulled into the instance via composer?

[samlikins]

Incorporated ClaimsToOEMR package and verified proper tie-ins.

[samlikins]

Removed PQRS SQL directory Added sites PQRS X12N837 inbox Added GIT ignore file to sites PQRS directory for importer log file Added importer configurations to globals Added new globals to importer configuration Added importer table to database file Rewrote importer script to utilize ClaimsToOEMR package

[samlikins]

Finding error while processing files:

2016/11/27 20:36:35 [error] 3783#0: *106 FastCGI sent in stderr: "PHP message: PHP Fatal error: Uncaught exception 'PDOException' with message 'SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '0' for key 'PRIMARY'' in /home/vagrant/openemr/vendor/illuminate/database/Connection.php:457

The x12_partners table doesn't have an auto_increment field, will need to update ClaimsToOEMR project to determine correct id field value, or update table to utilize auto_increment, before moving forward.

@aethelwulffe How do you suggest moving forward?

[samlikins]

Based upon @aethelwulffe's suggestion, I updated the EMR database to include auto_increment to the id (primary key) field for addresses, insurance_companies, phone_numbers, and x12_partners; doing this worked and should not provide any issue for segments of code that use the GenId functionality of ADODB.

[samlikins]

Encountered issue with PHP CLI taking 120 to initiate running code, eventually tracked issue down to the Xdebug PHP module; once deactivated, scripts ran immediately.

[aethelwulffe]

So is overall performance significantly higher than what we were seeing at first?

[samlikins]

Added file upload form to importer, verified process works.

[samlikins]

@aethelwulffe: sadly after reviewing the process, it doesn't seem to improve the performance any; will still need to determine a method to separate processes for digesting the X12N837 files.

[aethelwulffe]

I doubt Phalanger or Hip-Hop allows you to compile PHP+JS, and bitcode would be my first option as compile times usually eat up PHP cycles. Next would be an analysis for massive repetitive SQL calls when iterating over arrays instead of using IN clauses. Finally, and our most likely option will be using a separate box to run it on. Next question: Has this been running on Apache and maria? Have we checked it out there? Currently PQRS_Gateway is not up to PHP7 compatibility, and there are INNODB changes, so the environment we are testing it in needs to be checked out in production. Next, have we run it with significant resources (meaning maxing out per-script memory etc... on a dedicated machine) to see what its hard limits are? Child processes and "faking" multithreading could really help too. Have you considered Pthreads, or have you been using popen for forking processes?

[samlikins]

Made interface updates and moved PQRS associated files to appropriate location

[samlikins]

@aethelwulffe:

I'm not sure about Phalanger, and havn't had any experience compiling with Hip-Hop, but HHVM should have no issues running the code. As a note, PHP+JS would compile down the same way PHP+HTML would; the <?php and ?> tags turn on and off the compilation respectively, as so anything outside the PHP block would just be output as string (as usual).

I'm not exactly sure what you're referring to when you say "analysis for massive repetitive SQL calls when iterating over arrays instead of using IN clauses".

I agree, utilizing a separate system (tweaked for this job) to perform the consumption and loading of the X12N837 files would be the best bet.

My testing environment is using nginx and mysql; the server does little to reduce the speed, but running the code as a stand alone would not be hampered by a server, and the usage of MariaDB would only aid in the speed of it. The version of PHP that runs the DATA IMPORTER could be PHP 7, which would additionally aid to the speed of the processing. However, I'm not sure how INNODB changes would affect the system.

Load testing is most certainly a good idea, once it's been fully tested and implemented in a standard way, to make sure the structure is sane. As far as child process forking is concerned, I've not implemented any kind thus far and is my next area of approach. I am, however, looking at a slightly more sane angle for parallel and queue processing. I've been looking at Gearman, which has been used significantly on enterprise domains for resource and processing distribution with great success.

My next steps are to take the current PQRS/Issue-23 branch and merge it back into PQRS/master, then create a new branch for implementing the Gearman approach. This should also prove useful in distributed tasks for many things in the future.

[aethelwulffe]

I like the plan to merge and re-branch. Seems like the sane thing to do. I don't believe it touches anything that is used for production, and we should get it running in a production server environment for testing purposes.

The SQL bit was just my brain running over items that make things move slowly, like too many overly-specific calls to the database vs. returning arrays and the like. I wasn't pinging on anything specific at all.

Digging into any new (to us) technologies like compilers seem like a last resort vs. optimizations you already have in mind.

We already have PHP7 and MYSQL tweak "ways and means", as well as INNODB specific issues and how to resolve them from the LibreEHR project. If this comes to mind as a need, I am pretty sure I can make a mass of changes across the platform. That said, I think I would like to start an experimental branch on a LibreEHR fork to see how practical moving the base over is. We could benefit from the tabs layout greatly. There is also the fact that LibreEHR only has 45% of the files and 60% of the size of even the current OpenEMR, much less what it was months ago. I would like to be working off the same code base for everything, seeing as how the trash has been leaving that trailer.

On 12/5/2016 12:40 PM, Sam Likins wrote:

@aethelwulffe https://github.com/aethelwulffe:

I'm not sure about Phalanger, and havn't had any experience compiling with Hip-Hop, but HHVM should have no issues running the code. As a note, PHP+JS would compile down the same way PHP+HTML would; the |<?php| and |?>| tags turn on and off the compilation respectively, as so anything outside the PHP block would just be output as string (as usual).

I'm not exactly sure what you're referring to when you say "analysis for massive repetitive SQL calls when iterating over arrays instead of using IN clauses".

I agree, utilizing a separate system (tweaked for this job) to perform the consumption and loading of the X12N837 files would be the best bet.

My testing environment is using nginx and mysql; the server does little to reduce the speed, but running the code as a stand alone would not be hampered by a server, and the usage of MariaDB would only aid in the speed of it. The version of PHP that runs the DATA IMPORTER could be PHP 7, which would additionally aid to the speed of the processing. However, I'm not sure how INNODB changes would affect the system.

Load testing is most certainly a good idea, once it's been fully tested and implemented in a standard way, to make sure the structure is sane. As far as child process forking is concerned, I've not implemented any kind thus far and is my next area of approach. I am, however, looking at a slightly more sane angle for parallel and queue processing. I've been looking at Gearman, which has been used significantly on enterprise domains for resource and processing distribution with great success.

My next steps are to take the current PQRS/Issue-23 branch and merge it back into PQRS/master, then create a new branch for implementing the Gearman approach. This should also prove useful in distributed tasks for many things in the future.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/SunCoastConnection/PQRS_Gateway/issues/23#issuecomment-264921936, or mute the thread https://github.com/notifications/unsubscribe-auth/AAhzF_AdXpsc5Zoxk4Qqbymsv9OwHtkgks5rFEyggaJpZM4JrVh2.

[samlikins]

Branch PQRS/Issue-23 merged into PQRS/master

[samlikins]

@aethelwulffe:

Do you see any reason not to close this issue?

[samlikins]

@aethelwulffe:

I agree with your last post, with exception to using Gearman as the means to solve the parallel issue.

[leebc]

So I was going to say: Before you merge this back into PQRS/master, let's make a release branch that we're certain runs clean and has no conflicts...

[samlikins]

@leebc:

Well the common way to set a commit as a specific version is to set a tag to a commit. What version number should we use for this release, we're already at v4.2.0?

[leebc]

Well, that's openEMR 4.2.0... I was thinking PQRS Gateway v1.0 or 1.5 or v2.0. @aethelwulffe?

[samlikins]

I agree, @aethelwulffe should we be deleting all the OEMR specific branches, removing their tags, and starting anew for PQRS Gateway? @leebc: I feel v1.0 would be a good start for production, and as @aethelwulffe was suggesting about integrating LibreEHR, once that is done, we should make that v2.0.

[samlikins]

Here is a good help document on creating releases on GitHub.

[aethelwulffe]

More like .01.

1.0 sounds fine.

Mind you, I am pretty sure Sam's drop in bit won't affect anything else, but lets...just...test. I think there is an upgrade script for the SQL in any case, and that would have to be checked out.

On 12/5/2016 1:54 PM, leebc wrote:

Well, that's openEMR 4.2.0... I was thinking PQRS Gateway v1.0 or 1.5 or v2.0. @aethelwulffe https://github.com/aethelwulffe?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/SunCoastConnection/PQRS_Gateway/issues/23#issuecomment-264941871, or mute the thread https://github.com/notifications/unsubscribe-auth/AAhzF0Ciu1IA3abhYNDU2zmBsE9KI-fkks5rFF3LgaJpZM4JrVh2.

[samlikins]

Additionally, this can be done on the command line with git tag, and I think this may be the only way to tag a commit not at the head of a branch.

[samlikins]

I was wrong, you can create a release draft @ a specific commit. I've created a release draft, if anyone is interested in reviewing it.

[aethelwulffe]

Yeah, release tag like we should have been doing on LibreHealth.

We don't need any OEMR specific branches. No chance of those being useful.

On 12/5/2016 1:57 PM, Sam Likins wrote:

I agree, @aethelwulffe https://github.com/aethelwulffe should we be deleting all the OEMR specific branches, removing their tags, and starting anew for PQRS Gateway? @leebc https://github.com/leebc: I feel v1.0 would be a good start for production, and as @aethelwulffe https://github.com/aethelwulffe was suggesting about integrating LibreEHR, once that is done, we should make that v2.0.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/SunCoastConnection/PQRS_Gateway/issues/23#issuecomment-264942863, or mute the thread https://github.com/notifications/unsubscribe-auth/AAhzFxV0RNN9T9gG21feV8f8cEueyUlmks5rFF6pgaJpZM4JrVh2.

[aethelwulffe]

This is pretty much a dead project, though the repo is still here. Perhaps we might dust it off again in the future, but it looks like a much more direct linear reading of the files using a method similar to how we parse 835 files should be used. The values should be read into a database row for each segment, then implemented as inserts. Individual rows and transaction number can be checked to see if they are updates, so the more elaborate method of creating each datatype class and processing files in a queue with worker threads seems to not be necessary.