search

Shopify / pyoozie

Library for querying and scheduling with Apache Oozie
https://py-oozie.readthedocs.io
MIT License
11 stars 12 forks source link

Can we increase the Oozie WorkflowDefinitionMaxLength? #33

Open cfournie opened 7 years ago

cfournie commented 7 years ago

What happens if we increase the size of oozie.service.WorkflowAppService.WorkflowDefinitionMaxLength in oozie-default.xml? Can we construct workflwos greater than 100kb in size?

Currently, when creating a large workflow we get the following error (from /usr/local/oozie-4.1.0/logs/oozie.log):

org.apache.oozie.command.CommandException: E0736: Workflow definition length [109,985] exceeded maximum allowed length [100,000]
        at org.apache.oozie.command.wf.SubmitXCommand.execute(SubmitXCommand.java:272)
        at org.apache.oozie.command.wf.SubmitXCommand.execute(SubmitXCommand.java:75)
        at org.apache.oozie.command.XCommand.call(XCommand.java:281)
        at org.apache.oozie.DagEngine.submitJobFromCoordinator(DagEngine.java:139)
        at org.apache.oozie.command.coord.CoordActionStartXCommand.execute(CoordActionStartXCommand.java:185)
        at org.apache.oozie.command.coord.CoordActionStartXCommand.execute(CoordActionStartXCommand.java:63)
        at org.apache.oozie.command.XCommand.call(XCommand.java:281)
        at org.apache.oozie.command.XCommand.call(XCommand.java:351)
        at org.apache.oozie.command.coord.CoordActionReadyXCommand.execute(CoordActionReadyXCommand.java:121)
        at org.apache.oozie.command.coord.CoordActionReadyXCommand.execute(CoordActionReadyXCommand.java:41)
        at org.apache.oozie.command.XCommand.call(XCommand.java:281)
        at org.apache.oozie.command.XCommand.call(XCommand.java:351)
        at org.apache.oozie.command.coord.CoordActionInputCheckXCommand.execute(CoordActionInputCheckXCommand.java:230)
        at org.apache.oozie.command.coord.CoordActionInputCheckXCommand.execute(CoordActionInputCheckXCommand.java:68)
        at org.apache.oozie.command.XCommand.call(XCommand.java:281)
        at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:323)
        at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:252)
        at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:174)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.oozie.workflow.WorkflowException: E0736: Workflow definition length [109,985] exceeded maximum allowed length [100,000]
        at org.apache.oozie.service.WorkflowAppService.readDefinition(WorkflowAppService.java:129)
        at org.apache.oozie.service.LiteWorkflowAppService.parseDef(LiteWorkflowAppService.java:45)
        at org.apache.oozie.command.wf.SubmitXCommand.execute(SubmitXCommand.java:164)
        ... 20 more
cfournie commented 7 years ago

Changed /usr/local/oozie-4.1.0/conf/oozie-default.xml from 100000 to 200000 for WorkflowDefinitionMaxLength as follows:

    <property>
        <name>oozie.service.WorkflowAppService.WorkflowDefinitionMaxLength</name>
        <value>200000</value>
        <description>
            The maximum length of the workflow definition in bytes
            An error will be reported if the length exceeds the given maximum
        </description>
    </property>

When submitting another large workflow, the same error arises:

org.apache.oozie.command.CommandException: E0736: Workflow definition length [109,985] exceeded maximum allowed length [100,000]

Changing WorkflowDefinitionMaxLength in oozie-default.xml appear to be ineffectual.

cfournie commented 7 years ago

Added the following to /usr/local/oozie-4.1.0/conf/oozie-site.xml:

    <property>
        <name>oozie.service.WorkflowAppService.WorkflowDefinitionMaxLength</name>
        <value>200000</value>
        <description>
            The maximum length of the workflow definition in bytes
            An error will be reported if the length exceeds the given maximum
        </description>
    </property>

When submitting another large workflow, a new error arises:

2017-03-28 17:11:55,150 ERROR SubmitXCommand:538 - SERVER[oozie] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000000-170328171057119-oozie-root-C] ACTION[0000000-170328171057119-oozie-root-C@49] XException,
org.apache.oozie.command.CommandException: E0803: IO error, java.io.UTFDataFormatException: encoded string too long: 165335 bytes
        at org.apache.oozie.command.wf.SubmitXCommand.execute(SubmitXCommand.java:278)
        at org.apache.oozie.command.wf.SubmitXCommand.execute(SubmitXCommand.java:75)
        at org.apache.oozie.command.XCommand.call(XCommand.java:281)
        at org.apache.oozie.DagEngine.submitJobFromCoordinator(DagEngine.java:139)
        at org.apache.oozie.command.coord.CoordActionStartXCommand.execute(CoordActionStartXCommand.java:185)
        at org.apache.oozie.command.coord.CoordActionStartXCommand.execute(CoordActionStartXCommand.java:63)
        at org.apache.oozie.command.XCommand.call(XCommand.java:281)
        at org.apache.oozie.command.XCommand.call(XCommand.java:351)
        at org.apache.oozie.command.coord.CoordActionReadyXCommand.execute(CoordActionReadyXCommand.java:121)
        at org.apache.oozie.command.coord.CoordActionReadyXCommand.execute(CoordActionReadyXCommand.java:41)
        at org.apache.oozie.command.XCommand.call(XCommand.java:281)
        at org.apache.oozie.command.XCommand.call(XCommand.java:351)
        at org.apache.oozie.command.coord.CoordActionInputCheckXCommand.execute(CoordActionInputCheckXCommand.java:230)
        at org.apache.oozie.command.coord.CoordActionInputCheckXCommand.execute(CoordActionInputCheckXCommand.java:68)
        at org.apache.oozie.command.XCommand.call(XCommand.java:281)
        at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:323)
        at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:252)
        at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:174)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: java.io.UTFDataFormatException: encoded string too long: 165335 bytes
        at org.apache.oozie.util.WritableUtils.toByteArray(WritableUtils.java:51)
        at org.apache.oozie.WorkflowJobBean.setWorkflowInstance(WorkflowJobBean.java:351)
        at org.apache.oozie.command.wf.SubmitXCommand.execute(SubmitXCommand.java:203)
        ... 20 more
Caused by: java.io.UTFDataFormatException: encoded string too long: 165335 bytes
        at java.io.DataOutputStream.writeUTF(DataOutputStream.java:364)
        at java.io.DataOutputStream.writeUTF(DataOutputStream.java:323)
        at org.apache.oozie.workflow.lite.NodeDef.writeVersionOne(NodeDef.java:231)
        at org.apache.oozie.workflow.lite.NodeDef.write(NodeDef.java:263)
        at org.apache.oozie.workflow.lite.LiteWorkflowApp.write(LiteWorkflowApp.java:123)
        at org.apache.oozie.workflow.lite.LiteWorkflowInstance.write(LiteWorkflowInstance.java:597)
        at org.apache.oozie.util.WritableUtils.toByteArray(WritableUtils.java:46)
        ... 22 more

Changing WorkflowDefinitionMaxLength in oozie-site.xml is effectual, but a new issue arises.

kmtaylor-github commented 7 years ago

UTFDataFormatException: encoded string too long: 165335 bytes

Java's conception of UTF-8 has a 2-byte length field, thus capping encoded string length at 65535. https://docs.oracle.com/javase/7/docs/api/java/io/DataOutput.html#writeUTF(java.lang.String)

This raises another question: why can we get close to 100,000 (which is >64K)

kmtaylor-github commented 7 years ago

How did you create your testcase? The 64k limitation might be an undocumented restriction on any single element of a workflow.xml file, rather than the entire text.

cfournie commented 7 years ago

How did you create your testcase?

Poorly. I added a massive configuration block to a workflow.

This raises another question: why can we get close to 100,000 (which is >64K)

Oozie will break the definition up into smaller chunks. Still following the code to see why it wouldn't break up my workflow, but I suspect that my massive config block might be at fault. A large workflow with multiple actions may fare better.