Dear all,

I'm currently trying to publish documentation of a python project to a confluence instance, but the process keeps aborting with a HTTP 500 error from the server. Can you help me diagnose the root cause?

Thanks!

Output from the Report Command

(system)
 platform: Darwin-20.6.0-x86_64-i386-64bit
   python: 3.6.2 (default, Jun 21 2021, 15:27:31) [GCC Apple LLVM 12.0.0 (clang-1200.0.32.29)]
   sphinx: 4.2.0
 requests: 2.26.0
  builder: 1.6.0

(configuration)
confluence_page_hierarchy: True
confluence_parent_page: (set)
confluence_publish: True
confluence_publish_dryrun: False
confluence_server_pass: (set)
confluence_server_url: https://<removed>
confluence_server_user: (set)
confluence_space_name: (set; upper)

(error loading configuration)
unable to load configuration

(confluence instance)
 connected: no

Partial Output of the Build

I used the command sphinx-build -b confluence doc doc/_build/confluence -E -a -vvv -T to start a build.

writing output... [100%] space-migration
[app] emitting event: 'doctree-resolved'(<document: <target...><section "migrating one or more confluence sp ...>, 'space-migration')

publishing documents... [  8%] index
[app] emitting event: 'build-finished'(ConfluenceBadApiError('---\nAn unsupported Confluence API call has been made.\n\nREQ: PUT\nRSP: 500

Traceback (most recent call last):
  File "/projectdir/venv/lib/python3.6/site-packages/sphinx/cmd/build.py", line 280, in build_main
    app.build(args.force_all, filenames)
  File "/projectdir/venv/lib/python3.6/site-packages/sphinx/application.py", line 337, in build
    self.builder.build_all()
  File "/projectdir/venv/lib/python3.6/site-packages/sphinx/builders/__init__.py", line 257, in build_all
    self.build(None, summary=__('all source files'), method='all')
  File "/projectdir/venv/lib/python3.6/site-packages/sphinx/builders/__init__.py", line 360, in build
    self.finish()
  File "/projectdir/venv/lib/python3.6/site-packages/sphinxcontrib/confluencebuilder/builder.py", line 601, in finish
    self.publish_doc(docname, output)
  File "/projectdir/venv/lib/python3.6/site-packages/sphinxcontrib/confluencebuilder/builder.py", line 458, in publish_doc
    uploaded_id = self.publisher.storePage(title, data, parent_id)
  File "/projectdir/venv/lib/python3.6/site-packages/sphinxcontrib/confluencebuilder/publisher.py", line 535, in storePage
    self._updatePage(page, page_name, data, parent_id=parent_id)
  File "/projectdir/venv/lib/python3.6/site-packages/sphinxcontrib/confluencebuilder/publisher.py", line 742, in _updatePage
    self.rest_client.put('content', page['id'], updatePage)
  File "/projectdir/venv/lib/python3.6/site-packages/sphinxcontrib/confluencebuilder/rest.py", line 186, in put
    raise ConfluenceBadApiError(errdata)
sphinxcontrib.confluencebuilder.exceptions.ConfluenceBadApiError: ---
An unsupported Confluence API call has been made.

REQ: PUT
RSP: 500
URL: https://confluence-instance.example.com/rest/api
API: content
DATA: {
  "statusCode": 500,
  "message": "",
  "reason": "Internal Server Error"
}

Hey @nausicaea,

Assuming that the server URL is configured properly (it looks to be listed as https://confluence-instance.example.com/..., but it is assumed that this has been manually sanitized -- mentioning it, just in case), such an error is either due to an issue with content generated by this extension or an issue with the Confluence instance. Unfortunately, it is not always straight forward what it is. But there are approaches that can be made.

If it is data related, there is an undocumented data trace call which can be set:

confluence_adv_trace_data = True

When running with this option, a trace.log file will be generated in the working directory of the call and will output the raw contents of data into a file for each document that is processed. As documents are processed, generated output will be added into this file. Based on the above output, it is assuming that there might be some output in the index file which Confluence is not accepting. You could try to examine the contents of the last [data] entry in the trace log to look for what might be causing this issue. Another approach is to hack out the respective restructuredText content from the document to see what may be causing the issue (e.g. improper/unsupported characters, bug in the output of this extension, etc.).

Other possible issues may be related to the connection of the instance publishing documentation to the target Confluence instance. Assuming confluence_server_url is properly set, other issues this may be:

1) If the environment uses a proxy and the Confluence instance you are connecting to should not being going through a proxy, you may need to ensure your environment's no_proxy option is configured appropriately. 2) Another case may be that the target Confluence instance may be serviced through a reserve proxy and requires additional header data to process requests (see also #441). However, only your system administrators of your instance would know if this is a requirement. If custom header options are needed to communicate with the configured instance, options could be set via the confluence_publish_headers option.

A crude sanity check you could try is manually browsing the Confluence REST API endpoint from your browser. For example, if the configured Confluence URL was https://confluence-instance.example.com/, the following page:

https://confluence-instance.example.com/rest/api/search

Should output something like the following in a browser:

{"statusCode":400,"data":{"authorized":false,"valid":true,"errors":[],"successful":false},"message":"com.atlassian.confluence.api.service.exceptions.BadRequestException: cql query parameter is required"}

Dear @jdknight, I'm sorry I haven't replied yet. I haven't had the chance to test it out because other work issues had higher priority. I'll see to it that I can try out your suggestions this week, and then I'll get back to you.

I applied the data trace call option and looked at the last [data] entry in the trace.log file. As far as I was able to see, the code looked fine. But to be sure, I copied the contents in to the Confluence Source Editor of an empty page, and the content renders exactly as I would want it to, when saving the page. I followed the following steps:

Create an empty page in Confluence on my personal space
Save the page
Edit the page and click on "Open in source editor" in the upper right-hand corner of the page editor
Copy the contents of the last [data] section in trace.log directly into the source editor
Click on "Apply" and then on "Update" to save the page

Based on this experiment, I'm assuming that your extension correctly generates the Confluence storage format content from the restructuredText files.

Regarding the Confluence instance URLs: I did check the URL and the parameter confluence_server_url multiple times, and it is correct. We do serve Confluence behind an Apache2 proxy, but that hasn't caused any problems so far. We don't use a WAF or anything similar on the other hand. I am the "deputy" service administrator, but I'm not aware of any special headers we employ for authentication. Thanks for pointing that out though!

We also use the REST API for other things and it works as documented by Atlassian. But to be sure, I navigated to the Search API endpoint, and I'm getting exactly the unauthorized response you're suggesting.

I will try to publish the documentation by directly using the local Confluence server (circumventing the proxy): I'll post an update later.

Unfortunately, running the confluence builder plugin directly on our dev server didn't fix the issue. I'm still getting the same internal server error as before.

However, issuing a direct authenticated call to the REST API results in the expected response:

curl --basic -u <username> http://localhost:8090/rest/api/content/search?cql=space=AI

{"results":[{"id":"60130826","type":"comment","status":"current","title":"Re: CollabInfrastruktur.png","restrictions":{},"_links":{"webui":"/display/AI/01+Infrastruktur?preview=%2F60129512%2F60130490%2F1%2F60130826%2FCollabInfrastruktur.png","self":"https://intranet-dev.unibas.ch/rest/api/content/60130826"},"_expandable":{"container":"","metadata":"","extensions":"","operations":"","children":"","history":"/rest/api/content/60130826/history","ancestors":"","body":"","version":"","descendants":"","space":"/rest/api/space/AI"}},{"id":"60130099","type":"comment","status":"current","title":"Re: Wiki-Konzept","restrictions":{},"_links":{"webui":"/display/AI/Wiki-Konzept?focusedCommentId=60130099#comment-60130099","self":"https://intranet-dev.unibas.ch/rest/api/content/60130099"},"_expandable":{"container":"","metadata":"","extensions":"","operations":"","children":"","history":"/rest/api/content/60130099/history","ancestors":"","body":"","version":"","descendants":"","space":"/rest/api/space/AI"}},{"id":"60130097","type":"comment","status":"current","title":"Re: Wiki-Konzept","restrictions":{},"_links":{"webui":"/display/AI/Wiki-Konzept?focusedCommentId=60130097#comment-60130097","self":"https://intranet-dev.unibas.ch/rest/api/content/60130097"},"_expandable":{"container":"","metadata":"","extensions":"","operations":"","children":"","history":"/rest/api/content/60130097/history","ancestors":"","body":"","version":"","descendants":"","space":"/rest/api/space/AI"}},{"id":"60130100","type":"comment","status":"current","title":"Re: Wiki-Konzept","restrictions":{},"_links":{"webui":"/display/AI/Wiki-Konzept?focusedCommentId=60130100#comment-60130100","self":"https://intranet-dev.unibas.ch/rest/api/content/60130100"},"_expandable":{"container":"","metadata":"","extensions":"","operations":"","children":"","history":"/rest/api/content/60130100/history","ancestors":"","body":"","version":"","descendants":"","space":"/rest/api/space/AI"}},{"id":"60130732","type":"comment","status":"current","title":"Re: Bitte keine Permalink-URL zur Verlinkung innerhalb ITS-Wiki verwenden","restrictions":{},"_links":{"webui":"/display/AI/2021/01/26/Bitte+keine+Permalink-URL+zur+Verlinkung+innerhalb+ITS-Wiki+verwenden?focusedCommentId=60130732#comment-60130732","self":"https://intranet-dev.unibas.ch/rest/api/content/60130732"},"_expandable":{"container":"","metadata":"","extensions":"","operations":"","children":"","history":"/rest/api/content/60130732/history","ancestors":"","body":"","version":"","descendants":"","space":"/rest/api/space/AI"}},{"id":"60130734","type":"comment","status":"current","title":"Re: 8: Confluence Admin Users","restrictions":{},"_links":{"webui":"/display/AI/8%3A+Confluence+Admin+Users?focusedCommentId=60130734#comment-60130734","self":"https://intranet-dev.unibas.ch/rest/api/content/60130734"},"_expandable":{"container":"","metadata":"","extensions":"","operations":"","children":"","history":"/rest/api/content/60130734/history","ancestors":"","body":"","version":"","descendants":"","space":"/rest/api/space/AI"}},{"id":"60129773","type":"comment","status":"current","title":"Re: Atlassian Product + Pricing Plan Change Feb 2021","restrictions":{},"_links":{"webui":"/pages/viewpage.action?pageId=60129507&focusedCommentId=60129773#comment-60129773","self":"https://intranet-dev.unibas.ch/rest/api/content/60129773"},"_expandable":{"container":"","metadata":"","extensions":"","operations":"","children":"","history":"/rest/api/content/60129773/history","ancestors":"","body":"","version":"","descendants":"","space":"/rest/api/space/AI"}},{"id":"60129776","type":"comment","status":"current","title":"Re: Atlassian Product + Pricing Plan Change Feb 2021","restrictions":{},"_links":{"webui":"/pages/viewpage.action?pageId=60129507&focusedCommentId=60129776#comment-60129776","self":"https://intranet-dev.unibas.ch/rest/api/content/60129776"},"_expandable":{"container":"","metadata":"","extensions":"","operations":"","children":"","history":"/rest/api/content/60129776/history","ancestors":"","body":"","version":"","descendants":"","space":"/rest/api/space/AI"}},{"id":"60130825","type":"comment","status":"current","title":"Re: CollabInfrastruktur.png","restrictions":{},"_links":{"webui":"/display/AI/01+Infrastruktur?preview=%2F60129512%2F60130490%2F1%2F60130825%2FCollabInfrastruktur.png","self":"https://intranet-dev.unibas.ch/rest/api/content/60130825"},"_expandable":{"container":"","metadata":"","extensions":"","operations":"","children":"","history":"/rest/api/content/60130825/history","ancestors":"","body":"","version":"","descendants":"","space":"/rest/api/space/AI"}},{"id":"60129577","type":"comment","status":"current","title":"Re: Top 5 Makros","restrictions":{},"_links":{"webui":"/display/AI/Top+5+Makros?focusedCommentId=60129577#comment-60129577","self":"https://intranet-dev.unibas.ch/rest/api/content/60129577"},"_expandable":{"container":"","metadata":"","extensions":"","operations":"","children":"","history":"/rest/api/content/60129577/history","ancestors":"","body":"","version":"","descendants":"","space":"/rest/api/space/AI"}},{"id":"60129832","type":"comment","status":"current","title":"Re: git.urz.unibas.ch","restrictions":{},"_links":{"webui":"/display/AI/git.urz.unibas.ch?focusedCommentId=60129832#comment-60129832","self":"https://intranet-dev.unibas.ch/rest/api/content/60129832"},"_expandable":{"container":"","metadata":"","extensions":"","operations":"","children":"","history":"/rest/api/content/60129832/history","ancestors":"","body":"","version":"","descendants":"","space":"/rest/api/space/AI"}},{"id":"60130098","type":"comment","status":"current","title":"Re: Wiki-Konzept","restrictions":{},"_links":{"webui":"/display/AI/Wiki-Konzept?focusedCommentId=60130098#comment-60130098","self":"https://intranet-dev.unibas.ch/rest/api/content/60130098"},"_expandable":{"container":"","metadata":"","extensions":"","operations":"","children":"","history":"/rest/api/content/60130098/history","ancestors":"","body":"","version":"","descendants":"","space":"/rest/api/space/AI"}},{"id":"60130106","type":"comment","status":"current","title":"Re: 99 - OLD - Technische Dokumentation zu urz-git.urz.unibas.ch","restrictions":{},"_links":{"webui":"/display/AI/99+-+OLD+-+Technische+Dokumentation+zu+urz-git.urz.unibas.ch?focusedCommentId=60130106#comment-60130106","self":"https://intranet-dev.unibas.ch/rest/api/content/60130106"},"_expandable":{"container":"","metadata":"","extensions":"","operations":"","children":"","history":"/rest/api/content/60130106/history","ancestors":"","body":"","version":"","descendants":"","space":"/rest/api/space/AI"}},{"id":"60129901","type":"comment","status":"current","title":"Re: Atlassian Product + Pricing Plan Change Feb 2021","restrictions":{},"_links":{"webui":"/pages/viewpage.action?pageId=60129507&focusedCommentId=60129901#comment-60129901","self":"https://intranet-dev.unibas.ch/rest/api/content/60129901"},"_expandable":{"container":"","metadata":"","extensions":"","operations":"","children":"","history":"/rest/api/content/60129901/history","ancestors":"","body":"","version":"","descendants":"","space":"/rest/api/space/AI"}},{"id":"60130102","type":"comment","status":"current","title":"Re: 99 - OLD - Technische Dokumentation zu urz-git.urz.unibas.ch","restrictions":{},"_links":{"webui":"/display/AI/99+-+OLD+-+Technische+Dokumentation+zu+urz-git.urz.unibas.ch?focusedCommentId=60130102#comment-60130102","self":"https://intranet-dev.unibas.ch/rest/api/content/60130102"},"_expandable":{"container":"","metadata":"","extensions":"","operations":"","children":"","history":"/rest/api/content/60130102/history","ancestors":"","body":"","version":"","descendants":"","space":"/rest/api/space/AI"}},{"id":"60129974","type":"comment","status":"current","title":"Re: Collaboration Infrastruktur Home","restrictions":{},"_links":{"webui":"/display/AI/Collaboration+Infrastruktur+Home?focusedCommentId=60129974#comment-60129974","self":"https://intranet-dev.unibas.ch/rest/api/content/60129974"},"_expandable":{"container":"","metadata":"","extensions":"","operations":"","children":"","history":"/rest/api/content/60129974/history","ancestors":"","body":"","version":"","descendants":"","space":"/rest/api/space/AI"}},{"id":"60130107","type":"comment","status":"current","title":"Re: 99 - OLD - Technische Dokumentation zu urz-git.urz.unibas.ch","restrictions":{},"_links":{"webui":"/display/AI/99+-+OLD+-+Technische+Dokumentation+zu+urz-git.urz.unibas.ch?focusedCommentId=60130107#comment-60130107","self":"https://intranet-dev.unibas.ch/rest/api/content/60130107"},"_expandable":{"container":"","metadata":"","extensions":"","operations":"","children":"","history":"/rest/api/content/60130107/history","ancestors":"","body":"","version":"","descendants":"","space":"/rest/api/space/AI"}},{"id":"60130108","type":"comment","status":"current","title":"Re: 99 - OLD - Technische Dokumentation zu urz-git.urz.unibas.ch","restrictions":{},"_links":{"webui":"/display/AI/99+-+OLD+-+Technische+Dokumentation+zu+urz-git.urz.unibas.ch?focusedCommentId=60130108#comment-60130108","self":"https://intranet-dev.unibas.ch/rest/api/content/60130108"},"_expandable":{"container":"","metadata":"","extensions":"","operations":"","children":"","history":"/rest/api/content/60130108/history","ancestors":"","body":"","version":"","descendants":"","space":"/rest/api/space/AI"}},{"id":"60130047","type":"comment","status":"current","title":"Re: Working with Office Documents","restrictions":{},"_links":{"webui":"/display/AI/Working+with+Office+Documents?focusedCommentId=60130047#comment-60130047","self":"https://intranet-dev.unibas.ch/rest/api/content/60130047"},"_expandable":{"container":"","metadata":"","extensions":"","operations":"","children":"","history":"/rest/api/content/60130047/history","ancestors":"","body":"","version":"","descendants":"","space":"/rest/api/space/AI"}},{"id":"60130110","type":"comment","status":"current","title":"Re: 99 - OLD - Technische Dokumentation zu urz-git.urz.unibas.ch","restrictions":{},"_links":{"webui":"/display/AI/99+-+OLD+-+Technische+Dokumentation+zu+urz-git.urz.unibas.ch?focusedCommentId=60130110#comment-60130110","self":"https://intranet-dev.unibas.ch/rest/api/content/60130110"},"_expandable":{"container":"","metadata":"","extensions":"","operations":"","children":"","history":"/rest/api/content/60130110/history","ancestors":"","body":"","version":"","descendants":"","space":"/rest/api/space/AI"}},{"id":"60130049","type":"comment","status":"current","title":"Re: Working with Office Documents","restrictions":{},"_links":{"webui":"/display/AI/Working+with+Office+Documents?focusedCommentId=60130049#comment-60130049","self":"https://intranet-dev.unibas.ch/rest/api/content/60130049"},"_expandable":{"container":"","metadata":"","extensions":"","operations":"","children":"","history":"/rest/api/content/60130049/history","ancestors":"","body":"","version":"","descendants":"","space":"/rest/api/space/AI"}},{"id":"60129934","type":"comment","status":"current","title":"Re: Atlassian Product + Pricing Plan Change Feb 2021","restrictions":{},"_links":{"webui":"/pages/viewpage.action?pageId=60129507&focusedCommentId=60129934#comment-60129934","self":"https://intranet-dev.unibas.ch/rest/api/content/60129934"},"_expandable":{"container":"","metadata":"","extensions":"","operations":"","children":"","history":"/rest/api/content/60129934/history","ancestors":"","body":"","version":"","descendants":"","space":"/rest/api/space/AI"}},{"id":"60130083","type":"comment","status":"current","title":"Re: Testseite Plugins und Macros","restrictions":{},"_links":{"webui":"/display/AI/Testseite+Plugins+und+Macros?focusedCommentId=60130083#comment-60130083","self":"https://intranet-dev.unibas.ch/rest/api/content/60130083"},"_expandable":{"container":"","metadata":"","extensions":"","operations":"","children":"","history":"/rest/api/content/60130083/history","ancestors":"","body":"","version":"","descendants":"","space":"/rest/api/space/AI"}},{"id":"60130015","type":"comment","status":"current","title":"Re: 01 Infrastruktur","restrictions":{},"_links":{"webui":"/display/AI/01+Infrastruktur?focusedCommentId=60130015#comment-60130015","self":"https://intranet-dev.unibas.ch/rest/api/content/60130015"},"_expandable":{"container":"","metadata":"","extensions":"","operations":"","children":"","history":"/rest/api/content/60130015/history","ancestors":"","body":"","version":"","descendants":"","space":"/rest/api/space/AI"}},{"id":"60130079","type":"comment","status":"current","title":"Re: wiki.urz.unibas.ch - Wiki Migration","restrictions":{},"_links":{"webui":"/display/AI/wiki.urz.unibas.ch+-+Wiki+Migration?focusedCommentId=60130079#comment-60130079","self":"https://intranet-dev.unibas.ch/rest/api/content/60130079"},"_expandable":{"container":"","metadata":"","extensions":"","operations":"","children":"","history":"/rest/api/content/60130079/history","ancestors":"","body":"","version":"","descendants":"","space":"/rest/api/space/AI"}}],"start":0,"limit":25,"size":25,"cqlQuery":"space=AI","searchDuration":51,"totalSize":328,"_links":{"self":"https://intranet-dev.unibas.ch/rest/api/content/search?cql=space=AI","next":"/rest/api/content/search?limit=25&start=25&cql=space=AI","base":"https://intranet-dev.unibas.ch","context":""}}

I'm sending you the full output of my command line. Maybe you'll see something that I am not seeing:

sphinx-build -b confluence doc doc/_build/confluence -E -a -vvv > output.log 2>&1

output.log trace.log

I think I found the reason. First off, thank you for all the detailed information, it for sure helpful.

Inspecting the logs show this is part of the publish request:

  "id": "60129330",
  ...
  "ancestors": [
    {
      "id": "60129330"
    }
  ]

It is trying to update page with the same identifier that it is also trying to set the page as a child of. On Confluence Cloud, this is reported as a 400 error with a description message "Cannot add an existing ancestor as a child!"; but on (some) Confluence self-hosted solutions, it appears to give the non-descriptive 500 error.

It appears to me that you have root document titled Confluence Migration Assistant (CMA), which is trying to publish as a child of itself on the space. This is most likely due to confluence_parent_page being set to the same value. I believe you want to either do one of two things:

1) Instead of using confluence_parent_page, switch to confluence_publish_root instead. Although, this would require using the Confluence generated page identifier (observed from the URL or in the page's properties on Confluence) instead of a string value -- but with this you will be guaranteed it will be stored under a specific document (although, not flexible if you are using multiple spaces/languages base points; depending on the environment). For example (based on the output files provided):

confluence_publish_root = 60129330

2) (recommended) Remove the option confluence_parent_page completely. If you already have a page name on the Confluence instance created, any updates will update the existing document and child pages (with confluence_page_hierarchy being set) will be still stored inside the page. Moving this base page around in a space (if the space has mixed content) should not cause any issues with update, where it will still respect maintaining updates within the base page ( without getting into title issues; and some corner cases with orphan pages).

If you try any of these changes and it does not work, please let me know.

On that note, there are improvements that can be made to this extension. I will try to implement something to detect this use case and provide a more presentable error message to users. This also relates to #348 -- where there is for sure room for improvement in dealing with pre-detecting more error cases related to other nested page cases.

Ah! Now I understand! Thank you so much! I ended up opting for solution 1 and things worked out perfectly! :)

If you find time, I'd be grateful if you could detect self-referencing parent-child relationships.

sphinx-contrib / confluencebuilder

Unsupported API Call Exception on Build Publishing #517

Output from the Report Command

Partial Output of the Build