Shopify / pyoozie

Library for querying and scheduling with Apache Oozie
https://py-oozie.readthedocs.io
MIT License
11 stars 12 forks source link

Decrease coordinator schema version from 0.5 to 0.4 #13

Closed cfournie closed 7 years ago

cfournie commented 7 years ago

Submitting a coordinator to oozie 4.1.0 using xmlns="uri:oozie:coordinator:0.5" fails with:

2017-02-21 15:07:32,845  WARN V1JobsServlet:544 - SERVER[oozie] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] URL[POST http://localhost:11000/oozie/v2/jobs] error[E0701], E0701: XML schema error, cvc-elt.1.a: Cannot find the declaration of element 'coordinator-app'.
org.apache.oozie.servlet.XServletException: E0701: XML schema error, cvc-elt.1.a: Cannot find the declaration of element 'coordinator-app'.

Out of the box it can't read uri:oozie:coordinator:0.5. To fix this, this PR downgrads the schema that its generated coordinators use to uri:oozie:coordinator:0.4. There are only slight differences in available tags (none of which we use):

$ diff oozie-coordinator-0.5.xsd oozie-coordinator-0.4.xsd
10c10
<
---
>
12c12
<
---
>
19,20c19,20
< <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:coordinator="uri:oozie:coordinator:0.5"
<            elementFormDefault="qualified" targetNamespace="uri:oozie:coordinator:0.5">
---
> <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:coordinator="uri:oozie:coordinator:0.4"
>            elementFormDefault="qualified" targetNamespace="uri:oozie:coordinator:0.4">
35d34
<             <xs:element name="input-logic" type="coordinator:INPUTLOGIC" minOccurs="0" maxOccurs="1"/>
94,96c93
<         <xs:choice minOccurs="1" maxOccurs="1">
<             <xs:element name="and" type="coordinator:LOGICALAND" minOccurs="0" maxOccurs="1"/>
<             <xs:element name="or" type="coordinator:LOGICALOR" minOccurs="0" maxOccurs="1"/>
---
>         <xs:sequence minOccurs="1" maxOccurs="1">
98,142c95
<         </xs:choice>
<     </xs:complexType>
<     <xs:complexType name="INPUTLOGIC">
<         <xs:choice minOccurs="0" maxOccurs="unbounded">
<             <xs:element name="and" type="coordinator:LOGICALAND" minOccurs="0" maxOccurs="unbounded"/>
<             <xs:element name="or" type="coordinator:LOGICALOR" minOccurs="0" maxOccurs="unbounded"/>
<             <xs:element name="combine" type="coordinator:COMBINE" minOccurs="0" maxOccurs="unbounded"/>
<             <xs:element name="data-in" type="coordinator:LOGICALDATAIN" minOccurs="1" maxOccurs="unbounded"/>
<         </xs:choice>
<     </xs:complexType>
<     <xs:complexType name="LOGICALAND">
<         <xs:choice minOccurs="0" maxOccurs="unbounded">
<             <xs:element name="and" type="coordinator:LOGICALAND" minOccurs="0" maxOccurs="unbounded"/>
<             <xs:element name="or" type="coordinator:LOGICALOR" minOccurs="0" maxOccurs="unbounded"/>
<             <xs:element name="data-in" type="coordinator:LOGICALDATAIN" minOccurs="1" maxOccurs="unbounded"/>
<             <xs:element name="combine" type="coordinator:COMBINE" minOccurs="0" maxOccurs="unbounded"/>
<         </xs:choice>
<         <xs:attribute name="name" type="xs:string" use="optional"/>
<         <xs:attribute name="min" type="xs:string" use="optional"/>
<         <xs:attribute name="wait" type="xs:string" use="optional"/>
<     </xs:complexType>
<     <xs:complexType name="LOGICALOR">
<         <xs:choice minOccurs="0" maxOccurs="unbounded">
<             <xs:element name="and" type="coordinator:LOGICALAND" minOccurs="0" maxOccurs="unbounded"/>
<             <xs:element name="or" type="coordinator:LOGICALOR" minOccurs="0" maxOccurs="unbounded"/>
<             <xs:element name="data-in" type="coordinator:LOGICALDATAIN" minOccurs="1" maxOccurs="unbounded"/>
<             <xs:element name="combine" type="coordinator:COMBINE" minOccurs="0" maxOccurs="unbounded"/>
<         </xs:choice>
<         <xs:attribute name="name" type="xs:string" use="optional"/>
<         <xs:attribute name="min" type="xs:string" use="optional"/>
<         <xs:attribute name="wait" type="xs:string" use="optional"/>
<     </xs:complexType>
<     <xs:complexType name="COMBINE">
<         <xs:choice minOccurs="0" maxOccurs="unbounded">
<             <xs:element name="data-in" type="coordinator:LOGICALDATAIN" minOccurs="2" maxOccurs="unbounded"/>
<         </xs:choice>
<         <xs:attribute name="name" type="xs:string" use="optional"/>
<         <xs:attribute name="min" type="xs:string" use="optional"/>
<         <xs:attribute name="wait" type="xs:string" use="optional"/>
<     </xs:complexType>
<     <xs:complexType name="LOGICALDATAIN">
<         <xs:attribute name="name" type="xs:string" use="optional"/>
<         <xs:attribute name="min" type="xs:string" use="optional"/>
<         <xs:attribute name="wait" type="xs:string" use="optional"/>
<         <xs:attribute name="dataset" type="xs:string" use="required"/>
---
>         </xs:sequence>
165a119
>         <xs:attribute name="nocleanup" type="xs:boolean" use="optional"/>

review/ @kmtaylor-github @honkfestival

honkfestival commented 7 years ago

Could you add a supported versions section to the README? Otherwise LGTM

kmtaylor-github commented 7 years ago

Is s/coordinator/workflow an issue? Presumably we're unable to use any 0.5 artifacts.

cfournie commented 7 years ago

Is s/coordinator/workflow an issue? Presumably we're unable to use any 0.5 artifacts.

oozie-site.xml states that the schemas that it's using in Oozie 4.1.0 are:

    <property>
        <name>oozie.service.SchemaService.wf.ext.schemas</name>
        <value>
            shell-action-0.1.xsd,shell-action-0.2.xsd,shell-action-0.3.xsd,email-action-0.1.xsd,email-action-0.2.xsd,
            hive-action-0.2.xsd,hive-action-0.3.xsd,hive-action-0.4.xsd,hive-action-0.5.xsd,sqoop-action-0.2.xsd,
            sqoop-action-0.3.xsd,sqoop-action-0.4.xsd,ssh-action-0.1.xsd,ssh-action-0.2.xsd,distcp-action-0.1.xsd,
            distcp-action-0.2.xsd,oozie-sla-0.1.xsd,oozie-sla-0.2.xsd
        </value>
    </property>

We use shell-action-0.3 and email-action-0.2. at the moment, which are OK. And it accepted a oozie-workflow-0.5 workflow in my integration test.

cfournie commented 7 years ago

@kmtaylor-github 👍 / 👎 ?