Error when logging in to Wekan REST API when using Sandstorm Wekan

[xet7]

@kentonv

The problem is, on Standalone Wekan I can login to Wekan REST API using Wekan username and password, but on Sandstorm Wekan when I login with Google Auth, I get error with Sandstorm Webkey when I use Webkey as password. I don't know what other password I would use.

I tried this way:

1) On Wekan board create Webkey like in image by @ertanalytics https://github.com/wekan/wekan/wiki/Wekan-Sandstorm-cards-to-CSV-using-Python

Sure by using Python BeautifulSoup etc Wekan html and javascript could probably be parsed and used like in code above, but using REST API would be much nicer.

BTW @ertanalytics when I tried to use your code this way:

sudo apt install python-pip python-bs4 python-selenium python-pandas \
python-bson chromium-chromedriver chromium-browser-l10n

sudo pip install selenium --upgrade

sudo pip install splinter

And copied chromedriver from https://sites.google.com/a/chromium.org/chromedriver/home to my /usr/bin , I got this error, do you know solution to it?

$ python cards-to-csv.py 
Traceback (most recent call last):
  File "cards-to-csv.py", line 62, in <module>
    script = soup.find("script", attrs={'type' : 'text/inject-data'}).children.next()
AttributeError: 'NoneType' object has no attribute 'children'

2) I tried to login to Wekan REST API on Sandstorm:

curl https://USERNAME:APIKEY@api-ADDRESS.SOMETHING.sandcats.io/users/login \
     -d "username=USERNAME&password=APIKEY"

3) I get curl response:

{
  "error": "internal-server-error",
  "reason": "Internal server error"
}

4) On Sandstorm Wekan board console I get this error:

Error: Unauthorized [Unauthorized]
    at Object.Authentication.checkUserId (server/authentication.js:6:21)
    at models/users.js:563:20
    at packages/simple_json-routes.js:98:9

5) But if I try some other password, I get:

{
  "error": 400,
  "reason": "Match failed"
}

Using Wekan REST API on Sandstorm could make easier:

• uploading data from scripts etc to Sandstorm Wekan
• have mobile apps that use Sandstorm Wekan
• integrations with Huginn
• incoming Webhooks, etc.

Is there some documentation how with Meteor app like Wekan it would be possible to access info outside of grain, for example so that when importing attachments from Trello, some Sandstorm-compatible way could be used to download Trello attachments to Wekan?

I have not yet tried could EtherCalc API also be used on Sandstorm.

[ocdtrekkie]

I noticed that the Python script pulls from Selenium. I get the impression he is actually instructing Selenium to open the full Wekan grain and operating it in an automated fashion. It's definitely vastly preferable to query the API.

This is probably the documentation you're looking for, if you haven't seen this already: https://docs.sandstorm.io/en/latest/developing/http-apis/

[ertanalytics]

It looks like it last worked on 9/1/2017 and after the credentials were put in it was showing the full json text data of the wekan board and the subsequent steps parsed that json data.

The error:

Traceback (most recent call last): File "cards-to-csv.py", line 62, in <module> script = soup.find("script", attrs={'type' : 'text/inject-data'}).children.next() AttributeError: 'NoneType' object has no attribute 'children'

Is related to the parser not finding what it is expecting to start the parsing. Any ideas to navigate to the json data should fix this problem. Please let me know and I will update my code. I am trying a few different paths to see if I can figure out a way to get back to the json data with card titles etc. I would like all the board data if possible for the export.

@ocdtrekkie - I am just instructing Selenium to open the URL. Once the page loads I grab the background data and immediately close the browser:

`browser2 = Browser('chrome')

browser2.visit(apiURLnoAuth) # Fill in the url

sleep(1) #Time in seconds

soup = BeautifulSoup(browser2.html,'html.parser') browser2.quit()`

Once the data is assigned to the 'soup' variable, I am done with Selenium.

[xet7]

@ertanalytics

There is already a way to Import and Export Wekan board to and from JSON at Sandstorm Wekan, when you click board menu (hamburger icon, 3 lines) and click there Import or Export. But I don't know how to do this with API.

Trello import on Sandstorm Wekan does not import attachments yet, because Wekan does not have Sandstorm compatible access to outside of grain. Importing Trello attachments works on Standalone Wekan (like Docker etc) and then you can export Wekan JSON that includes attachments in JSON as Base64 encoded, and then import that Wekan JSON to Sandstorm with attachments in JSON.

[ertanalytics]

@xet7 - This appears to be broken as well in Sandstorm. The download starts, but fails 'Failed - Server problem' while download backup on Sandstorm continues to work.

On 9/1/2017 and earlier, visiting the API URL had the export data readily available and that was what the script I wrote was parsing.

[xet7]

@ertanalytics

What version of Wekan you have on Sandstorm? Have you updated to latest? I have 0.47.0

When updating, also update all your grains.

Please include your Sandstorm Wekan grain console logs so I can see exact errors.

[ertanalytics]

Version: 0.47.0

SANDSTORM SUPERVISOR: Grain shutdown requested. SANDSTORM SUPERVISOR: Starting up grain. Sandbox type: userns Starting Mongo... 2017-10-07T18:58:45.562+0000 I STORAGE Engine custom option: log=(prealloc=false,file_max=200KB) about to fork child process, waiting until server is ready for connections. forked process: 10 child process started successfully, parent exiting Starting Meteor... { [Error: Cannot find module '../build/Release/bson'] code: 'MODULE_NOT_FOUND' } js-bson: Failed to load c++ bson extension, using pure JS version Error: 59d92409a78c710200228a12 does not exist at Object.Future.wait (/programs/server/node_modules/fibers/future.js:449:15) at packages/meteor.js:213:24 at models/export.js:85:15 at Array.map (native) at Exporter.build (models/export.js:80:60) at models/export.js:32:62 at packages/simple_json-routes.js:98:9

---

at /programs/server/npm/node_modules/meteor/cfs_gridfs/node_modules/mongodb/lib/mongodb/gridfs/gridstore.js:200:22
at /programs/server/npm/node_modules/meteor/cfs_gridfs/node_modules/mongodb/lib/mongodb/collection/query.js:164:5
at /programs/server/npm/node_modules/meteor/cfs_gridfs/node_modules/mongodb/lib/mongodb/cursor.js:778:35
at Cursor.close (/programs/server/npm/node_modules/meteor/cfs_gridfs/node_modules/mongodb/lib/mongodb/cursor.js:1009:5)
at Cursor.nextObject (/programs/server/npm/node_modules/meteor/cfs_gridfs/node_modules/mongodb/lib/mongodb/cursor.js:778:17)
at commandHandler (/programs/server/npm/node_modules/meteor/cfs_gridfs/node_modules/mongodb/lib/mongodb/cursor.js:741:14)
at /programs/server/npm/node_modules/meteor/cfs_gridfs/node_modules/mongodb/lib/mongodb/db.js:1903:9
at Server.Base._callHandler (/programs/server/npm/node_modules/meteor/cfs_gridfs/node_modules/mongodb/lib/mongodb/connection/base.js:453:41)
at /programs/server/npm/node_modules/meteor/cfs_gridfs/node_modules/mongodb/lib/mongodb/connection/server.js:488:18
at [object Object].MongoReply.parseBody (/programs/server/npm/node_modules/meteor/cfs_gridfs/node_modules/mongodb/lib/mongodb/responses/mongo_reply.js:68:5)
at [object Object].<anonymous> (/programs/server/npm/node_modules/meteor/cfs_gridfs/node_modules/mongodb/lib/mongodb/connection/server.js:446:20)
at emitOne (events.js:77:13)
at [object Object].emit (events.js:169:7)
at [object Object].<anonymous> (/programs/server/npm/node_modules/meteor/cfs_gridfs/node_modules/mongodb/lib/mongodb/connection/connection_pool.js:207:13)
at emitTwo (events.js:87:13)
at [object Object].emit (events.js:172:7)

[xet7]

Is it only error with this Wekan grain?

Can you export some other Wekan grain?

[ertanalytics]

It is only happening to the one board.

Other wekan grains are exporting json correctly.

[xet7]

I added this info to Wekan wiki

Exporting Wekan board to JSON file with Bash script

1) On Wekan grain, get Webkey like this:

https://api-URL.SUBDOMAIN.sandcats.io#APIKEY

2) Modity URL, SUBDOMAIN and APIKEY to this Bash script that exports board to file directly:

curl https://Bearer:APIKEY@api-URL.SUBDOMAIN.sandcats.io/api/boards/sandstorm/export?authToken=#APIKEY > wekanboard.json

[kentonv]

It seems like you're successfully doing the things you need to do on Sandstorm to talk to a grain API. The errors are coming from Wekan, not Sandstorm.

With that said, it's likely that you'll want to change the way the Wekan API is authorized on Sandstorm. Sandstorm will automatically add headers to the API requests identifying the user -- the same headers it adds to all HTTP requests. https://docs.sandstorm.io/en/latest/developing/auth/#headers-that-an-app-receives

The X-Sandstorm-User-Id header in particular contains the Sandstorm identity ID, which matches the field services.sandstorm.id in the user record in Mongo. So you can figure out who the user is that way, instead of requiring a separate login step.

[ertanalytics]

As far as trying the Wekan API, I am not having any luck using the links through Sandstorm. The only thing that has worked was the export URL thus far. I am betting the directory structure inhibits the normal API request functions, however I am a bit green on API testing and might be missing something.

[ertanalytics]

I added a bounty to this issue to try to get links made for the API that also work in Sandstorm. Only the export link works and that is not part of the API:

https://github.com/wekan/wekan/wiki/REST-API

Bounty page: https://www.bountysource.com/teams/wekan/issues

https://www.bountysource.com/issues/50166298-error-when-logging-in-to-wekan-rest-api-when-using-sandstorm-wekan

[xet7]

I think I could try to fix this by exposing /api at sandstorm-pkgdef.capnp file when I get some free time.

[ertanalytics]

Wish I had the setup and understanding to do these changes myself. Any idea on timeline?

[xet7]

@ertanalytics

No idea. Maybe in near or distant future.

[ertanalytics]

Following up on this one. I have reached out to xet7 to assist and he is having trouble with his sandstorm based building VM. If anyone has a working development environment to test the REST API for sandstorm version and build from it to get it to production please let me know, I would like to get this feature as soon as possible.

[xet7]

@ertanalytics

As you can see from these IRC chatlogs, only @kentonv is currently able to build Sandstorm from source, because he made changes how Sandstorm was built: https://botbot.me/freenode/sandstorm/2018-02-11/?tz=Europe/Helsinki

I don't know are there pull request to documentation yet.

I also have not yet figured out all steps yet.

[xet7]

@ertanalytics

Probably these fixes to build script were recently merged: https://github.com/sandstorm-io/sandstorm/pull/3034

[xet7]

@ertanalytics

Add new issue to Sandstorm if you still have problems: https://github.com/sandstorm-io/sandstorm/issues

[xet7]

@ertanalytics

And also add info to this issue if you get Sandstorm built, and how you did it.

[xet7]

@ertanalytics

I added issue to Sandstorm, I still have problems. https://github.com/sandstorm-io/sandstorm/issues/3036

[xet7]

Install instructions included at Sandstorm repo should now work also for you. You can see changed merged pull requests about Docs fixes at https://github.com/sandstorm-io/sandstorm/pulls?q=is%3Apr+is%3Aclosed

So you can clone that repo and follow instructions. It should work OK.

[xet7]

Requirements: 1) Add cards - including specific labels, descriptions, checklists. 2) Move cards - from one list to another. 3) Remove members - from the entire board. 4) Change assignment - Assign certain members to certain cards or remove them.

[xet7]

Current data extraction is done with this: https://github.com/wekan/wekan/wiki/Wekan-Sandstorm-cards-to-CSV-using-Python

[xet7]

Progress: 1) Add cards - including specific labels, descriptions, checklists.

I tried adding apiPath to Wekan similar to this way: https://github.com/sandstorm-io/sandstorm-generic-rest-app/blob/master/sandstorm-pkgdef.capnp#L58-L59

Here is some examples of permissions and roles: https://docs.sandstorm.io/en/latest/developing/auth/#defining-permissions-and-roles

Wekan has a way to update existing card with REST API https://github.com/wekan/wekan/issues/1037#issuecomment-301271356 at Standalone Wekan like this:

PUT /api/boards/:boardId/lists/:listId/cards/:cardId

I did get board, list and card IDs from Sandstorm board exported Wekan JSON file.

Then I tried updating card this way:

curl -H "Authorization: Bearer APIKEY" \
     -H "Content-type:application/json" \
     -X PUT \
     https://Bearer:APIKEY@api-APIURL.SUBDOMAIN.sandcats.io/api/boards/sandstorm/lists/LIST-ID/cards/CARD-ID \
     -d '{ "title": "Testing title of card 1", "description": "Some description about card", "authorId": "WEKAN-USER-ID" }'

Then I get this error in Wekan grain debug log:

Error: Unauthorized [Unauthorized]
    at Object.Authentication.checkUserId (server/authentication.js:12:21)
    at models/cards.js:521:20
    at packages/simple_json-routes.js:98:9

This means that I did not get syntax errors anymore like with previous curl tests I did, so syntax is OK, but there could be some double login issue, or I just have not logged in properly yet.

[xet7]

Maybe it's just API authentication issue like @kentonv said at https://github.com/wekan/wekan/issues/1279#issuecomment-334987880 . I'll look at API authentication code.

[xet7]

About exporting Wekan board with API:

• at Sandstorm hosted Wekan that was accepted by @ocdtrekkie : export with API works
• on my own sandcats.io server: export with API does not work
• on local Sandstorm Wekan with meteor-dev: export with API does not work

When I disable that authentication code at server/authentication.js:12:21 above, then API just does not work. That means, that API does not give correct result like exporting board or changing card, it just gives back Wekan frontpage html source.

@kentonv

Do you have any idea why this happens? This makes even testing API hard.

There has been changes in Wekan that some of the API like checklists does not work, but it would be nice to even have API working locally.

I have also tried to uncommend admin permission code at: https://github.com/wekan/wekan/blob/devel/sandstorm-pkgdef.capnp#L182-L188 and also adding apiPath, but those did not make anything work.

I'm still new to all this how permissions at Sandstorm works, but I'm trying to figure this out, as it would help many Wekan users, including my clients, that are stuck at this currently.

[kentonv]

I get the "unauthorized" error on both Oasis and self-hosted. This looks like a problem on the app side.

How are you deciding if the request is authorized? You should probably be looking at the x-sandstorm-permissions header, or look up the user from x-sandstorm-user-id.

Here's the only code in Wekan which I can find that looks at either header:

https://github.com/wekan/wekan/blob/cb099d10933041d3c5814495e1dbf288e1d67ec6/sandstorm.js#L326-L340

Will the above set authorization as needed for API requests? If not, you may need to add similar code in the API implementation.

[ertanalytics]

Went ahead and upgraded to the latest and did some testing and got some consistent results and some strange, but still not functioning results. ` Create User Attempt 1:

curl -H "Authorization: Bearer PeOE8n2aH0hv7xY9k4AtOd5Yqe59Aqqk_uQjprfJv-F" \ -H "Content-type:application/json" \ -X POST \ https://api-001a314424eb592f4f48ada88108fd7f.ertanalytics.us/api/register \ -d '{ "username": "NewUser", "password": "mypassword", "email": "testUser12b@mailinator.com" }' Result: {"isClientSafe":true,"error":"Unauthorized","reason":"Unauthorized","message":"Unauthorized [Unauthorized]","errorType":"Meteor.Error","statusCode":401}

=======================================================================

Create User Attempt 2: curl -H "Authorization: Bearer PeOE8n2aH0hv7xY9k4AtOd5Yqe59Aqqk_uQjprfJv-F" \ -H "Content-type:application/json" \ -X POST \ https://Bearer:PeOE8n2aH0hv7xY9k4AtOd5Yqe59Aqqk_uQjprfJv-F@api-001a314424eb592f4f48ada88108fd7f.ertanalytics.us/api/users \ -d '{ "username": "NewUser", "password": "mypassword", "email": "testUser12b@mailinator.com" }' Result: {"isClientSafe":true,"error":"Unauthorized","reason":"Unauthorized","message":"Unauthorized [Unauthorized]","errorType":"Meteor.Error","statusCode":401}

=======================================================================

Create User Attempt 3: curl -H "Authorization: Bearer PeOE8n2aH0hv7xY9k4AtOd5Yqe59Aqqk_uQjprfJv-F" \ -H "Content-type:application/json" \ -X POST \ https://Bearer:PeOE8n2aH0hv7xY9k4AtOd5Yqe59Aqqk_uQjprfJv-F@api-001a314424eb592f4f48ada88108fd7f.ertanalytics.us/api/register \ -d '{ "username": "NewUser", "password": "mypassword", "email": "testUser12b@mailinator.com" }' Result: <!DOCTYPE html>

scriptBREAKINGTHISFORGITHUBPOSTtype="text/javascript" src="/2ca780339237d68d344bdbed5b5e07ac1ecc3a92.js?meteor_js_resource=true"/script ======================================================================= Create User Attempt 4: curl -H "Authorization: Bearer PeOE8n2aH0hv7xY9k4AtOd5Yqe59Aqqk_uQjprfJv-F" \ -H "Content-type:application/json" \ -X POST \ https://Bearer:PeOE8n2aH0hv7xY9k4AtOd5Yqe59Aqqk_uQjprfJv-F@api-001a314424eb592f4f48ada88108fd7f.ertanalytics.us#PeOE8n2aH0hv7xY9k4AtOd5Yqe59Aqqk_uQjprfJv-F/api/users?authToken=PeOE8n2aH0hv7xY9k4AtOd5Yqe59Aqqk_uQjprfJv-F \ -d '{ "username": "NewUser", "password": "mypassword", "email": "testUser12b@mailinator.com" }' Result: No Error, but nothing happens ======================================================================= Create User Attempt 5: curl -H "Authorization: Bearer PeOE8n2aH0hv7xY9k4AtOd5Yqe59Aqqk_uQjprfJv-F" \ -H "Content-type:application/json" \ -X POST \ https://Bearer:PeOE8n2aH0hv7xY9k4AtOd5Yqe59Aqqk_uQjprfJv-F@api-001a314424eb592f4f48ada88108fd7f.ertanalytics.us#PeOE8n2aH0hv7xY9k4AtOd5Yqe59Aqqk_uQjprfJv-F/api/register?authToken=PeOE8n2aH0hv7xY9k4AtOd5Yqe59Aqqk_uQjprfJv-F \ -d '{ "username": "NewUser", "password": "mypassword", "email": "testUser12b@mailinator.com" }' Result: No Error, but nothing happens ======================================================================= https://Bearer:PeOE8n2aH0hv7xY9k4AtOd5Yqe59Aqqk_uQjprfJv-F@api-001a314424eb592f4f48ada88108fd7f.ertanalytics.us/ `

[xet7]

@kentonv

Problem is still in authorization code at https://github.com/wekan/wekan/issues/1279#issuecomment-377693993 in wekan/server/authentication.js:12:21. I did try to add checking for sandstorm id, etc, and those checks seem to pass ok. But then comes the actual check of authorization place in that line 12, and I could not yet figure out how to make it happy with sandstorm id and pass that authorization check.

Somewhere else was some code comment, that Wekan at Sandstorm runs like admin user of standalone Wekan. I have not yet looked more closely about permissions implementation in Wekan and Sandstorm.

[ertanalytics]

Wanted to provide an update. Stand in process is now failing and has for some time at Sandstorm wekan pages around 19MB or less, only just noticed. Page source has gotten too big to be properly parsed as an alternative to this API. It has been dropping data on my board for some time and it looks like more than half the board is missing in the extracts.

Not sure if there is any way to further escalate this card and I appreciate the attention it has gotten thus far.

[xet7]

@ertanalytics

On short term, it's possible to export data to Standalone Wekan: https://github.com/wekan/wekan/wiki/Export-from-Wekan-Sandstorm-grain-.zip-file

In long term, there will be performance improvements to Wekan, those are at roadmap.

[xet7]

@ertanalytics

Yes, I should fix Sandstorm API, when I get some free time. Sorry it's taking so long.

[xet7]

@ertanalytics

What version of Wekan you have in Sandstorm? I may need to disable one feature that slows down Wekan.

[ertanalytics]

I am on sandstorm version 218

[xet7]

@ertanalytics

What Wekan version?

[xet7]

At Sandstorm / Apps / Wekan / Version

[ertanalytics]

I am on Version 0.94.0~2018-05-03 of Wekan

[xet7]

@ertanalytics

Can you test:

• make backup of all grains
• upgrade to newest Wekan
• is there any difference?

[ertanalytics]

While I do backup nightly, I am not in a position where I can comfortably restore the exact same version of my environment if I upgrade and I fear change like most without thorough testing. There is already a way to test on Sandstorm's Oasis. If no changes were made to the API structure in wekan or sandstorm, I believe this bug will persist. I did a test below, however I am not super confident of my formatting for API requests in sandstorm apps other than the already working export API get request.

https://oasis.sandstorm.io/

I tested again, not that I really know how the format should go in Sandstorm for user addition, but I tried this and got a 'Bad Request' for user add while I was able to confirm export was still working:

import requests import json

APIBegURL = 'https://Bearer:nS_0nDcc63vZ9QWpuNCkPK5BBMGLDiLxP-0JNhVKz79@api-a5587d9539ca906a2aa5e1eebc7dac89.oasis.sandstorm.io' EndURL = '#nS_0nDcc63vZ9QWpuNCkPK5BBMGLDiLxP-0JNhVKz79'

Commands

Confirm API still works for exporting entire board

exportCommand = '/api/boards/sandstorm/export?authToken=' APIRequest = APIBegURL + exportCommand + EndURL response = requests.get(APIRequest) print(response.text) ## Returns full export

Bad Request Result for User Addition

usersCommand = '/api/boards/sandstorm/users?authToken=' APIRequest = APIBegURL + usersCommand + EndURL response = requests.post(APIRequest, data='', json='{ "username": "tester", "password": "tester", "email": "tester@example.com", "fromAdmin": "true" }' ) print(response.text) ## Returns bad request

[ocdtrekkie]

@ertanalytics Did you test with 1.33 on Oasis or 1.34?

[ertanalytics]

Version 1.33.0~2018-08-16 @ocdtrekkie

[xet7]

@ertanalytics

Please test Wekan Sandstorm version v1.39, it has speedups, does it help enough with performance?

[ocdtrekkie]

Also, since you tested with the pre-SandstormApi change version, test out the API too and let me know if it behaves differently. I wanted to do some testing myself but I've been really busy of late and don't know how much time I have to devote to side projects at the moment.

[ertanalytics]

@xet7 , it is more about the API not working than performance. I have not noticed a performance increase with the update. My larger, more used board, takes about 23 seconds to respond when you click on the browser to view from being in the background. It is 20.8 MB board on Sandstorm.

@ocdtrekkie , I have updated to version 1.39 of Wekan on Sandstorm, updated Sandstorm with Dev channel and now the board export API above no longer works. Other API calls continue to not work. I get status OK response, but it just gives me the same response text no matter what API commands I put in: ` <!DOCTYPE html>

`

[xet7]

@ertanalytics

Export using Wekan board hamburger menu / Export still works, so API is enabled on on Wekan side. I know this, because if at Standalone Wekan WITH_API=false, Exporting board at Wekan menu does not work, but gives similar HTML output like you have above. That settings WITH_API=true is at code here: https://github.com/wekan/wekan/blob/devel/sandstorm-pkgdef.capnp#L240

I don't know what effect did addition of /api by @ocdtrekkie do: https://github.com/wekan/wekan/blob/devel/sandstorm-pkgdef.capnp#L229

@kentonv

Does Sandstorm API still work? Or did some update disable it?

[xet7]

I only tested at sandstorm.io though. I'll test at something.sandcats.io next.

[xet7]

I tested something.sandcats.io , created webkey etc, it's the same. Export using API does not work. Export using Wekan menu does work. Something prevents calling API.

[ocdtrekkie]

Hm..... what if you remove the /api from your request URL, @xet7?

I'm wondering if setting apiPath to /api makes it add it to the URL, which would be /api/api. If this is true, we can change the apiPath to / to fix this.

[xet7]

@ocdtrekkie

/api/api and /api/api/api did not work yet in rest api. I try modifying sandstorm-pkgdef.capnp file.