Document Packaging Service

[kohsah]

Documents once they complete the workflow cycle need to be packaged for publication onto the portal.

So the XML document + attachments are packaged into a ZIP document, and the ZIP document is posted onto a receiving service in the Portal which accepts the document.

The md5sum of the zip file is transmitted along with the zip file so the receiving service can verify the consistency of what has been received.

Zip file must contain --

• the Akoma Ntoso XML document
• Attachments with the folder structure

This services needs to be implemented in editor-fe, from where it can be invoked by the editor-ui. The service needs to be aware of the onward portal service that receives the documents, and must needs to respond back to the editor-ui with a response indicating the status of the packaging and transmission.

This need not be a real-time response since packaging, transmission and response [and the response will be guided by the response of the onward service based on whether it accepted the document or not]. So we need to implement a message queue - i.e. the editor-ui submitting a packaging request immediately receives a response from editor-fe stating the packaging and publishing request has been placed on a queue.

That queue needs to be processed by a back-end / process that checks the queue for new requests and processes them, and updates the status of the request in the queue. The editor-fe API subscribed to the queue gets updates, and sends responses back to the editor-ui.

I have used RabbitMQ and ActiveMQ in the past and RabbitMQ pretty much works out of the box, and has good API support across platforms. You define an exchange to publish messages into and it has a publisher api for adding messages and a consumer api for reading messages.

[kohsah]

Diagram indicating different components --

[kohsah]

@ashwinibm not clear to me what happens when you publish and the portal q processor isnt running.

I published without the portal-qprocessor running, got an error message in the editor qprocessor log:

[*] Waiting for messages. To exit press CTRL+C
 [x] iriQ: '/akn/mg/act/legge/1994-11-16/LOI_N°94-025/fra@/!main'
 IN: toPortal
 IN: loadXmlForIri
 IN: prepareZip
72776 total bytes
archiver has been finalized and the output file descriptor has closed.
 IN: postPkg
 IN: postToPortal
 IN: publishStatus
{ iri: '/akn/mg/act/legge/1994-11-16/LOI_N°94-025/fra@/!main',
  status: 'failed',
  message: 'Error on Portal Q Processor' }
 Status dispatched to Editor-FE
{ Error: connect ECONNREFUSED 127.0.0.1:9004
    at Object._errnoException (util.js:1022:11)
    at _exceptionWithHostPort (util.js:1044:20)
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1198:14)
  code: 'ECONNREFUSED',
  errno: 'ECONNREFUSED',
  syscall: 'connect',
  address: '127.0.0.1',
  port: 9004,
  config:
..

Subsequently when i started the qprocessor, it didnt indicate receiving any messages. The document was still under processing -- and the message lost in processing. In such scenarios the message needs to be requeued so 'posttoPortal' can be retried

[kohsah]

@ashwinibm "publish document" is currently a hard coded button. It needs to be a workflow rendered action like the other action buttons.

e.g. what publishDocument() does can be decomposed to what transitDocument() does... and the latter is the generic transitter. you don't need a separate back-end API for publish document, you just need a switch/case evaluator in the /gwc/workfows/transit in editor-fe ... That way the front-end does not need to change, the behavior is standard irrespective of what happens in the transition... and the transition backend api handler decides what to do based on the transition name. Otherwise in the below model if you change the workflow both the front-end and back-end will break.

    publishDocument() {
        const iri = docIri(this.props.pkg);
        console.log("PUBLISH DOCUMENT", iri);
        axios.post(apiUrl("document-publish"), {data: {iri}})
            .then( (response) => {
                // {"success":{"code":"publish_document","message":"/db/docs/gawati-client-data/akn/ke/act/legge/1970-06-03/akn_ke_act_legge_1970-06-03_Cap_44_eng_main.xml"}}
                const data = response.data;
                if (data.success) {
                    notifySuccess(T(`The document ${iri} has been submitted for publishing`));
                }
            })
            .catch( (err) => {
                console.log("publishDocument ", err);
            });
    } 

[kohsah]

@ashwinibm got an error in portal-publisher :

looks like a path join issue in publishPkg.js : notice the incorrect path, repatead D:\develop : D:\\develop\\github\\gawati\\gawati-portal-publisher\\tmp\\D:\\develop\\github\\gawati\\gawati-portal-qprocessor\\tmp\\tmpja0sh\\akn_mu_act_legge_1919-10-24_GN_212-1919_eng_main.xml

Extracted to  D:\develop\github\gawati\gawati-portal-qprocessor\tmp\tmpja0sh
 IN: publishStatus
{ iri: '/akn/mu/act/legge/1919-10-24/GN_212-1919/eng@/!main',
  status: 'failed',
  message: 'Error on Portal Publisher' }
 Status dispatched to Portal Q Processor
{ Error: ENOENT: no such file or directory, open 'D:\develop\github\gawati\gawati-portal-publisher\tmp\D:\develop\github\gawati\gawati-portal-qprocessor\tmp\tmpja0sh\akn_mu_act_legge_1919-10-24_GN_212-1919_eng_main.xml'
  errno: -4058,
  code: 'ENOENT',
  syscall: 'open',
  path: 'D:\\develop\\github\\gawati\\gawati-portal-publisher\\tmp\\D:\\develop\\github\\gawati\\gawati-portal-qprocessor\\tmp\\tmpja0sh\\akn_mu_act_legge_1919-10-24_GN_212-1919_eng_main.xml' }