commit request

[fhdk]

From documentation it says to post a form with

  /api/packages/commit:
    post:
      summary: Commit package transaction
      operationId: commitTransaction
      requestBody:
        required: true
        content:
          multipart/form-data:
            schema:
              type: object
              properties:
                packageFile:
                  type: string
                  format: binary
                packageSignature:
                  type: string
                  format: binary
                packageSection:
                  type: string

What is the packageSection expected to contain ?

Is there any url parameres which are undocumented ?

Example branch, repo, architecture

Responsees

      responses:
        "200":
          description: Transaction committed successfully
        "400":
          description: Invalid request
        "403":
          description: No permissions

There is one response missing in documentation for almost every endpoint

        "401":
          description: Unauthorized

I keep getting 500 - Internal Server Error when trying to post using the available documentation

In the web ui - dropping packages - one is expected to commit too.

Is there something special to account for when using the /api/packages/commit endpoint ?

[LordTermor]

packageSection

It should be a regular section JSON:

{
 "branch": "string",
 "repository": "string",
 "architecture": "string"
}

I'm not sure what is correct to reference it as a part of form-data

There is one response missing in documentation for almost every endpoint

Oh, right. Will put it.

[fhdk]

While you are at it - the response type in the documentation is either missing or set to text/html - the expected response on everything is json - right ?

[LordTermor]

While you are at it - the response type in the documentation is either missing or set to text/html - the expected response on everything is json - right ?

Yes, the response type is always json

[fhdk]

By experimenting with the web interface I realised a commit can consist of more than one package where each package consist of two binaries - pkg and sig.

From your comment above regarding the packageSection it is a valid json string with the target branch/repo/arch

So based on that I devised a form which in a test scenario gives

test_repo = os.path.join(os.path.dirname(__file__), 'repo')
files = {
    ("packageFile", (None, f"{test_repo}/abseil-cpp-20240116.2-2-x86_64.pkg.tar.zst")),
    ("packageSignature", (None, f"{test_repo}/abseil-cpp-20240116.2-2-x86_64.pkg.tar.zst.sig")),
    ("packageFile", (None, f"{test_repo}/arch-install-scripts-28-1-any.pkg.tar.zst")),
    ("packageSignature", (None, f"{test_repo}/arch-install-scripts-28-1-any.pkg.tar.zstsig")),
}
body = {"packageSection": {'branch': 'unstable', 'repository': 'extra', 'architecture': 'x86_64'}}
response = requests.request('post', 'https://httpbin.org/post', files=files, json=body)
print("response headers")
pprint(response.json()['headers'])
print("response json")
pprint(response.json())

result of test post using httpbin.org

  --> response headers
{'Accept': '*/*',
 'Accept-Encoding': 'gzip, deflate',
 'Content-Length': '733',
 'Content-Type': 'multipart/form-data; '
                 'boundary=14b5dce38d7080c416a855abef746b46',
 'Host': 'httpbin.org',
 'User-Agent': 'python-requests/2.32.3',
 'X-Amzn-Trace-Id': 'Root=1-66839e0e-13430d162741228c78e257e3'}
  --> response json
{'args': {},
 'data': '',
 'files': {},
 'form': {'packageFile': ['/a/projects/manjaro-bxt/bxtctl/repo/abseil-cpp-20240116.2-2-x86_64.pkg.tar.zst',
                          '/a/projects/manjaro-bxt/bxtctl/repo/arch-install-scripts-28-1-any.pkg.tar.zst'],
          'packageSignature': ['/a/projects/manjaro-bxt/bxtctl/repo/abseil-cpp-20240116.2-2-x86_64.pkg.tar.zst.sig',
                               '/a/projects/manjaro-bxt/bxtctl/repo/arch-install-scripts-28-1-any.pkg.tar.zstsig']},
 'headers': {'Accept': '*/*',
             'Accept-Encoding': 'gzip, deflate',
             'Content-Length': '733',
             'Content-Type': 'multipart/form-data; '
                             'boundary=14b5dce38d7080c416a855abef746b46',
             'Host': 'httpbin.org',
             'User-Agent': 'python-requests/2.32.3',
             'X-Amzn-Trace-Id': 'Root=1-66839e0e-13430d162741228c78e257e3'},
 'json': None,
 'origin': 'ip.x.y.z',
 'url': 'https://httpbin.org/post'}

Does this look like I am on right track ?

[fhdk]

This one gives me {"status": "ok"}.

test_repo = os.path.join(os.path.dirname(__file__), "repo")
files = {
    ("packageFile", (None, open(f"{test_repo}/abseil-cpp-20240116.2-2-x86_64.pkg.tar.zst", "rb"))),
    ("packageSignature", (None, open(f"{test_repo}/abseil-cpp-20240116.2-2-x86_64.pkg.tar.zst.sig", "rb"))),
}
body = {
    "packageSection": {
        "branch": "unstable",
        "repository": "extra",
        "architecture": "x86_64",
    }
}

headers = {"Authorization": f"Bearer {token}"}
req = requests.session()
req.headers.update(headers)
response = req.post("https://hostname/api/packages/commit", files=files, json=body, headers=headers)
print(response.text)

Login to the web portal - I expected to see the package - :grimacing: but no.

Most likely my own fault ... I need to look at the arguments for the request

[LordTermor]

the correct format for form is:

package{n}.section -> JSON package{n}.signatureFile -> Signature File package{n} (or anything starting with package{n}.) -> Package file

Where n is a number starting with 1 (doesn't matter what specific number is, just should match for all of fields of the same package).

You can check the TS code here: https://github.com/anydistro/bxt/blob/c0e83973a46fe02b02823ca4d27c146a28a06f0c/web/src/hooks/BxtHooks.ts#L101-L128

If you would have some ideas on how to properly represent it in the docs feel free to share your thoughts.

[fhdk]

Thank your for the hint.

It is getting better - now I get 502 - Bad Gatway

test_repo = os.path.join(os.path.dirname(__file__), "repo")
upload_form = {
    ("package1", open(f"{test_repo}/arch-install-scripts-28-1-any.pkg.tar.zst", "rb")),
    ("package1.signatureFile", open(f"{test_repo}/arch-install-scripts-28-1-any.pkg.tar.zst.sig", "rb")),
}
upload_target = {
    "branch": "unstable",
    "repository": "extra",
    "architecture": "x86_64"
}

pprint(upload_form)
headers = {"Authorization": f"Bearer {token}"}
req = requests.session()
req.headers.update(headers)
print("request begin --> ", time.strftime("%Y-%m-%d %H:%M:%S"))
response = req.post(endpoint, files=upload_form, json=upload_target)
print("response recv --> ", time.strftime("%Y-%m-%d %H:%M:%S"))
print("          headers ", response.headers)
print("          status  ", response.status_code)

/home/fh/.cache/pypoetry/virtualenvs/bxtctl-J8vlmj3L-py3.12/bin/python /a/projects/manjaro-bxt/bxtctl/testpad.py 
{('package1',
  <_io.BufferedReader name='/a/projects/manjaro-bxt/bxtctl/repo/arch-install-scripts-28-1-any.pkg.tar.zst'>),
 ('package1.signatureFile',
  <_io.BufferedReader name='/a/projects/manjaro-bxt/bxtctl/repo/arch-install-scripts-28-1-any.pkg.tar.zst.sig'>)}
request begin -->  2024-07-02 10:31:10
502
response recv -->  2024-07-02 10:32:11
          headers  {'Alt-Svc': 'h3=":443"; ma=2592000', 'Server': 'Caddy', 'Date': 'Tue, 02 Jul 2024 08:32:11 GMT', 'Content-Length': '0'}

If you would have some ideas on how to properly represent it in the docs feel free to share your thoughts.

Sure - when I know how to get it working :grin:

[fhdk]

From what I can figure out by banging the python documentation is that a multipart/form-data is expected to be binary.

package.section is a json-string - that cannot go into a multipart/form-data object - as it is not binary.

I will try and make a json blob sending it as part of the form - see where that goes

[fhdk]

new test code

test_repo = os.path.join(os.path.dirname(__file__), "repo")
upload_target = {
    "branch": "unstable",
    "repository": "extra",
    "architecture": "x86_64"
}
section = json.dumps(upload_target)
upload_form = {
    ("package1", open(f"{test_repo}/arch-install-scripts-28-1-any.pkg.tar.zst", "rb")),
    ("package1.signatureFile", open(f"{test_repo}/arch-install-scripts-28-1-any.pkg.tar.zst.sig", "rb")),
    ("package1.section", bytes(section, "utf-8"))
}

pprint(f"upload_form : {upload_form}")
pprint(f"section     : {upload_target}")
headers = {"Authorization": f"Bearer {token}"}
req = requests.session()
req.headers.update(headers)
print("request begin --> ", time.strftime("%Y-%m-%d %H:%M:%S"))
response = req.post(endpoint, files=upload_form, json=upload_target)
print("response recv --> ", time.strftime("%Y-%m-%d %H:%M:%S"))
print("          headers ", response.headers)
print("          status  ", response.status_code)

Still getting 502

('upload_form : {(\'package1.section\', b\'{"branch": "unstable", '
 '"repository": "extra", "architecture": "x86_64"}\'), '
 "('package1.signatureFile', <_io.BufferedReader "
 "name='/a/projects/manjaro-bxt/bxtctl/repo/arch-install-scripts-28-1-any.pkg.tar.zst.sig'>), "
 "('package1', <_io.BufferedReader "
 "name='/a/projects/manjaro-bxt/bxtctl/repo/arch-install-scripts-28-1-any.pkg.tar.zst'>)}")
("section     : {'branch': 'unstable', 'repository': 'extra', 'architecture': "
 "'x86_64'}")
request begin -->  2024-07-02 10:57:46
response recv -->  2024-07-02 10:58:47
          headers  {'Alt-Svc': 'h3=":443"; ma=2592000', 'Server': 'Caddy', 'Date': 'Tue, 02 Jul 2024 08:58:47 GMT', 'Content-Length': '0'}
          status   502

[fhdk]

Still 502

upload_form : 
{('package1.section', b"{'branch':'unstable','repository':'extra','architecture':'x86_64'}"), ('package1.signatureFile', <_io.BufferedReader name='/a/projects/manjaro-bxt/bxtctl/repo/arch-install-scripts-28-1-any.pkg.tar.zst.sig'>), ('package1', <_io.BufferedReader name='/a/projects/manjaro-bxt/bxtctl/repo/arch-install-scripts-28-1-any.pkg.tar.zst'>)}
request begin -->  2024-07-02 11:05:52
response recv -->  2024-07-02 11:06:53
          headers  {'Alt-Svc': 'h3=":443"; ma=2592000', 'Server': 'Caddy', 'Date': 'Tue, 02 Jul 2024 09:06:53 GMT', 'Content-Length': '0'}
          status   502

[LordTermor]

try

params = {
 "package1.section": '{"branch": "unstable", "repository": "extra", "architecture": "x86_64"}'
}

response = req.post(endpoint, files=upload_form, data=params)

[fhdk]

So what you reference as section, is passed in using query string ?

I understood the documentation in a way that each package or list of packages and the corresponding signature should be passed in using the form.

It makes sense - if it is so - in that it lines up with how the other requests are done where branch, repo and architecture is passed in using url parameters.

Your previous comment indicates that each package has a section reference - which would be required if a commit has two packages where each has it's own section

package{n}.section -> JSON package{n}.signatureFile -> Signature File package{n} (or anything starting with package{n}.) -> Package file

Where n is a number starting with 1 (doesn't matter what specific number is, just should match for all of fields of the same package).

If you are passing the section in via url parameters those would be for all packages in the commit - thus eliminating the need for package{n}.section ?

How do your web app pass it to the backend at /api/packages/commit ?

Since your documentation says multipart/form-data - you will have to generate a form - right ?

I think the main issue right now is the 502 status code I am getting.

upload_form : 
{('package1', <_io.BufferedReader name='/a/projects/manjaro-bxt/bxtctl/repo/arch-install-scripts-28-1-any.pkg.tar.zst'>), ('package1.signatureFile', <_io.BufferedReader name='/a/projects/manjaro-bxt/bxtctl/repo/arch-install-scripts-28-1-any.pkg.tar.zst.sig'>)}
params
{'packages1.section': {'branch': 'unstable', 'repository': 'extra', 'architecture': 'x86_64'}}
request begin -->  2024-07-02 13:11:05
response recv -->  2024-07-02 13:12:06
          headers  {'Alt-Svc': 'h3=":443"; ma=2592000', 'Server': 'Caddy', 'Date': 'Tue, 02 Jul 2024 11:12:06 GMT', 'Content-Length': '0'}
          status   502

upload_form : 
{('package1.file', <_io.BufferedReader name='/a/projects/manjaro-bxt/bxtctl/repo/arch-install-scripts-28-1-any.pkg.tar.zst'>), ('package1.signatureFile', <_io.BufferedReader name='/a/projects/manjaro-bxt/bxtctl/repo/arch-install-scripts-28-1-any.pkg.tar.zst.sig'>)}
params : 
{'packages1.section': '"{\\"branch\\": \\"unstable\\", \\"repository\\": \\"extra\\", \\"architecture\\": \\"x86_64\\"}"'}
request begin -->  2024-07-02 13:30:30
response recv -->  2024-07-02 13:31:31
          headers  {'Alt-Svc': 'h3=":443"; ma=2592000', 'Server': 'Caddy', 'Date': 'Tue, 02 Jul 2024 11:31:31 GMT', 'Content-Length': '0'}
          status   502

[fhdk]

I got really confused about the 502 - but I finally figure out how to format the form so the endpoint accepts it

upload_target = {
    "branch": "unstable",
    "repository": "extra",
    "architecture": "x86_64"
}
upload_form = {
    ("package1.file", ("arch-install-scripts-28-1-any.pkg.tar.zst", open(f"{test_repo}/arch-install-scripts-28-1-any.pkg.tar.zst", "rb"))),
    ("package1.signatureFile", ("arch-install-scripts-28-1-any.pkg.tar.zst", open(f"{test_repo}/arch-install-scripts-28-1-any.pkg.tar.zst.sig", "rb"))),
    ("package1.section", (None, json.dumps(upload_target))),
}

headers = {"Authorization": f"Bearer {token}"}

print("upload_form : ")
print(upload_form)
req = requests.session()
req.headers.update(headers)

print("request begin --> ", time.strftime("%Y-%m-%d %H:%M:%S"))
response = req.post(endpoint, files=upload_form)
print("response recv --> ", time.strftime("%Y-%m-%d %H:%M:%S"))
print("          headers ", response.headers)
print("          status  ", response.status_code)

Result

upload_form : 
{('package1.file', ('arch-install-scripts-28-1-any.pkg.tar.zst', <_io.BufferedReader name='/a/projects/manjaro-bxt/bxtctl/repo/arch-install-scripts-28-1-any.pkg.tar.zst'>)), ('package1.signatureFile', ('arch-install-scripts-28-1-any.pkg.tar.zst', <_io.BufferedReader name='/a/projects/manjaro-bxt/bxtctl/repo/arch-install-scripts-28-1-any.pkg.tar.zst.sig'>)), ('package1.section', (None, '{"branch": "unstable", "repository": "extra", "architecture": "x86_64"}'))}
params : 
{'package1.section': {'branch': 'unstable', 'repository': 'extra', 'architecture': 'x86_64'}}
{"package1.section": {"branch": "unstable", "repository": "extra", "architecture": "x86_64"}}
request begin -->  2024-07-02 14:20:34
response recv -->  2024-07-02 14:20:34
          headers  {'Alt-Svc': 'h3=":443"; ma=2592000', 'Content-Length': '15', 'Content-Type': 'application/json; charset=utf-8', 'Date': 'Tue, 02 Jul 2024 12:20:34 GMT', 'Server': 'Caddy, drogon/1.8.2'}
          status   200

And the package files land in the target branch/arch/repo

[fhdk]

With recent update to the commit endpoint - three new properties has been added.

                to_delete:
                  type: string
                to_move:
                  type: string
                to_copy:
                  type: string

What value(s) are these properties expected to contain ?

I assume the prefix is the same - is the content like section where a json object defines....

package1.to_delete =
package1.to_move =
package1.to_copy =

[LordTermor]

There are no prefixes in these three. They are JSON arrays to_delete is JSON of

[
{
 "name":"string", 
 "section": {"branch":"string", "repository":"string", "architecture":"string"}
}
]

to_copy and to_move are

[
{
 "name":"string", 
 "from_section": {"branch":"string", "repository":"string", "architecture":"string"}, 
 "to_section": {"branch":"string", "repository":"string", "architecture":"string"}
}
]

[fhdk]

But they are part of the form ?

When commiting the form at least one package is required - at least that is how I read your code in BxtHooks.ts

If bxtctl should be able to create a move or delete - that would not be possible without also commiting a package.

Perhaps it would be better with a separate endpoint(s) ?

[LordTermor]

When commiting the form at least one package is required - at least that is how I read your code in BxtHooks.ts

No, you can just provide to_delete, to_copy or to_move

[fhdk]

No, you can just provide to_delete, to_copy or to_move

That implies they are optional - right ?

I have open this again :frowning_face:

The documentation state (I inserted a comment)

  /api/packages/commit:
    post:
      summary: Commit package transaction
      operationId: commitTransaction
      requestBody:

// -------------- required is true
        required: true
        content:
          multipart/form-data:
            schema:
              type: object
              properties:
                packageFile:
                  type: string
                  format: binary
                packageSignature:
                  type: string
                  format: binary
                packageSection:
                  type: string
                to_delete:
                  type: string
                to_move:
                  type: string
                to_copy:
                  type: string

      responses:
        "200":
          description: Transaction committed successfully
        "400":
          description: Invalid request
        "403":
          description: No permissions

Sending the same form (the one the worked) - without the optionals - now results in 500

upload_form : 
{('package1.section', (None, '{"branch": "stable", "repository": "multilib", "architecture": "x86_64"}')), ('package1.signatureFile', ('arch-install-scripts-28-1-any.pkg.tar.zst.sig', <_io.BufferedReader name='/a/projects/manjaro-bxt/bxtctl/repo/arch-install-scripts-28-1-any.pkg.tar.zst.sig'>)), ('package1.file', ('arch-install-scripts-28-1-any.pkg.tar.zst', <_io.BufferedReader name='/a/projects/manjaro-bxt/bxtctl/repo/arch-install-scripts-28-1-any.pkg.tar.zst'>))}
request begin -->  2024-07-07 06:39:47
response recv -->  2024-07-07 06:39:47
          headers  {'Alt-Svc': 'h3=":443"; ma=2592000', 'Content-Length': '0', 'Content-Type': 'text/html; charset=utf-8', 'Date': 'Sun, 07 Jul 2024 04:39:47 GMT', 'Server': 'Caddy, drogon/1.8.2'}
          status   500

Adding in the elements 'optional' elements as empty json arrays makes the request valid.

upload_form : 
{('package1.section', (None, '{"branch": "stable", "repository": "multilib", "architecture": "x86_64"}')), ('package1.file', ('arch-install-scripts-28-1-any.pkg.tar.zst', <_io.BufferedReader name='/a/projects/manjaro-bxt/bxtctl/repo/arch-install-scripts-28-1-any.pkg.tar.zst'>)), ('to_copy', (None, '[]')), ('to_delete', (None, '[]')), ('package1.signatureFile', ('arch-install-scripts-28-1-any.pkg.tar.zst.sig', <_io.BufferedReader name='/a/projects/manjaro-bxt/bxtctl/repo/arch-install-scripts-28-1-any.pkg.tar.zst.sig'>)), ('to_move', (None, '[]'))}
request begin -->  2024-07-07 06:50:55
response recv -->  2024-07-07 06:50:56
          headers  {'Alt-Svc': 'h3=":443"; ma=2592000', 'Content-Length': '15', 'Content-Type': 'application/json; charset=utf-8', 'Date': 'Sun, 07 Jul 2024 04:50:56 GMT', 'Server': 'Caddy, drogon/1.8.2'}
          status   200

So the elements is not 'optonal' - even though you express it as being optional - given the comma separation and the or clause.

you can just provide to_delete, to_copy or to_move

How can one send a delete or copy or move request for an existing package as the request returns 500 if not all elements is present ?

[fhdk]

I tested the move request and expected the package uploaded to multilib to be moved to extra.

Am I misunderstanding your example ?

When uploading a package one need six form parts

• packageN.file
• packageN.signatureFile
• packageN.section
• to_delete
• do_move
• to_copy

When sending a move an element to_move is required, containing a json object array consisting of at least one named element with source and destination

Trying to move the package from above upload (arch-install-scripts)

upload_form : 
{('to_move',
  '[{"name": "arch-install-scripts-28-1-any.pkg.tar.zst", "from_section": '
  '{"branch": "stable", "repository": "multilib", "architecture": "x86_64"}, '
  '"to_section": {"branch": "stable", "repository": "extra", "architecture": '
  '"x86_64"}}]')}
request begin -->  2024-07-07 07:28:31
response recv -->  2024-07-07 07:28:31
          headers  {'Alt-Svc': 'h3=":443"; ma=2592000', 'Content-Length': '15', 'Content-Type': 'application/json; charset=utf-8', 'Date': 'Sun, 07 Jul 2024 05:28:31 GMT', 'Server': 'Caddy, drogon/1.8.2'}
          status   200

returns 200 but the package doesn't move.

[fhdk]

See #84