speckleworks / SpeckleRevitReboot

Check a brand new Speckle at: https://github.com/specklesystems
https://speckle.systems/
25 stars 13 forks source link

Unable to create Sender. Could not access custom SpeckleServer. #13

Closed htlcnn closed 4 years ago

htlcnn commented 4 years ago

I'm running a custom SpeckleServer on my virtual machine. I'm able to login to Web UI and create Projects, Streams there. I installed SpeckleRevit and use it with Revit 2018. I was able to create Sender and sent data to hestia server. For my custom SpeckleServer, I logged in successfully in SpeckleRevit, but when I added a new sender and choose that server, SpeckleRevit said Could not access that server (is it online?) or no server Selection. Is there any specific configuration to be able to send data to custom SpeckleServer? image

didimitrie commented 4 years ago

Hello @htlcnn , i think this might be related to https://github.com/speckleworks/SpeckleUi/issues/7 - is your local server https?

htlcnn commented 4 years ago

@didimitrie no, my server is not https. Is https required?

didimitrie commented 4 years ago

I'm afraid so - we will try to remove this limitation, or at least allow for localhost:xxxx urls.

htlcnn commented 4 years ago

Thanks. I'll add https to my server using Let's encrypt self-signed certificate.

htlcnn commented 4 years ago

@didimitrie I've added a self-signed certificate to my SpeckleServer. This time SpeckleRevit fails to find the server at the first step. It says There seems to be no speckle server there: image

Here's api response from https://10.15.0.30/api (json beautified):

{
    "isSpeckleServer": true,
    "serverName": "BIM UDIC SpeckleServer",
    "version": "v1.9.8",
    "api": {
        "projects": [{
            "method": "POST",
            "route": "https://10.15.0.30/api/projects"
        }, {
            "method": "GET",
            "route": "https://10.15.0.30/api/projects"
        }, {
            "method": "GET",
            "route": "https://10.15.0.30/api/projects/admin"
        }, {
            "method": "GET",
            "route": "https://10.15.0.30/api/projects/:projectId"
        }, {
            "method": "PUT",
            "route": "https://10.15.0.30/api/projects/:projectId"
        }, {
            "method": "PUT",
            "route": "https://10.15.0.30/api/projects/:projectId/addstream/:streamId"
        }, {
            "method": "DELETE",
            "route": "https://10.15.0.30/api/projects/:projectId/removestream/:streamId"
        }, {
            "method": "PUT",
            "route": "https://10.15.0.30/api/projects/:projectId/adduser/:userId"
        }, {
            "method": "DELETE",
            "route": "https://10.15.0.30/api/projects/:projectId/removeuser/:userId"
        }, {
            "method": "PUT",
            "route": "https://10.15.0.30/api/projects/:projectId/upgradeuser/:userId"
        }, {
            "method": "PUT",
            "route": "https://10.15.0.30/api/projects/:projectId/downgradeuser/:userId"
        }, {
            "method": "DELETE",
            "route": "https://10.15.0.30/api/projects/:projectId"
        }],
        "clients": [{
            "method": "POST",
            "route": "https://10.15.0.30/api/clients"
        }, {
            "method": "GET",
            "route": "https://10.15.0.30/api/clients"
        }, {
            "method": "GET",
            "route": "https://10.15.0.30/api/clients/:clientId"
        }, {
            "method": "PUT",
            "route": "https://10.15.0.30/api/clients/:clientId"
        }, {
            "method": "DELETE",
            "route": "https://10.15.0.30/api/clients/:clientId"
        }, {
            "method": "GET",
            "route": "https://10.15.0.30/api/streams/:streamId/clients"
        }],
        "streams": [{
            "method": "POST",
            "route": "https://10.15.0.30/api/streams"
        }, {
            "method": "GET",
            "route": "https://10.15.0.30/api/streams"
        }, {
            "method": "GET",
            "route": "https://10.15.0.30/api/streams/admin"
        }, {
            "method": "GET",
            "route": "https://10.15.0.30/api/streams/:streamId"
        }, {
            "method": "PUT",
            "route": "https://10.15.0.30/api/streams/:streamId"
        }, {
            "method": "DELETE",
            "route": "https://10.15.0.30/api/streams/:streamId"
        }, {
            "method": "POST",
            "route": "https://10.15.0.30/api/streams/:streamId/clone"
        }, {
            "method": "GET",
            "route": "https://10.15.0.30/api/streams/:streamId/diff/:otherId"
        }, {
            "method": "GET",
            "route": "https://10.15.0.30/api/streams/:streamId/delta/:otherId"
        }, {
            "method": "POST",
            "route": "https://10.15.0.30/api/streams/:streamId/delta"
        }, {
            "method": "GET",
            "route": "https://10.15.0.30/api/streams/:streamId/objects"
        }, {
            "method": "GET",
            "route": "https://10.15.0.30/api/streams/:streamId/clients"
        }],
        "accounts": [{
            "method": "POST",
            "route": "https://10.15.0.30/api/accounts/register"
        }, {
            "method": "POST",
            "route": "https://10.15.0.30/api/accounts/login"
        }, {
            "method": "GET",
            "route": "https://10.15.0.30/api/accounts"
        }, {
            "method": "GET",
            "route": "https://10.15.0.30/api/accounts/admin"
        }, {
            "method": "PUT",
            "route": "https://10.15.0.30/api/accounts"
        }, {
            "method": "GET",
            "route": "https://10.15.0.30/api/accounts/:userId"
        }, {
            "method": "PUT",
            "route": "https://10.15.0.30/api/accounts/:userId"
        }, {
            "method": "POST",
            "route": "https://10.15.0.30/api/accounts/search"
        }],
        "comments": [{
            "method": "GET",
            "route": "https://10.15.0.30/api/comments"
        }, {
            "method": "GET",
            "route": "https://10.15.0.30/api/comments/assigned"
        }, {
            "method": "POST",
            "route": "https://10.15.0.30/api/comments/:resourceType/:resourceId"
        }, {
            "method": "GET",
            "route": "https://10.15.0.30/api/comments/:resourceType/:resourceId"
        }, {
            "method": "GET",
            "route": "https://10.15.0.30/api/comments/:commentId"
        }, {
            "method": "PUT",
            "route": "https://10.15.0.30/api/comments/:commentId"
        }, {
            "method": "DELETE",
            "route": "https://10.15.0.30/api/comments/:commentId"
        }],
        "objects": [{
            "method": "GET",
            "route": "https://10.15.0.30/api/streams/:streamId/objects"
        }, {
            "method": "POST",
            "route": "https://10.15.0.30/api/objects"
        }, {
            "method": "POST",
            "route": "https://10.15.0.30/api/objects/derive"
        }, {
            "method": "GET",
            "route": "https://10.15.0.30/api/objects/:objectId"
        }, {
            "method": "POST",
            "route": "https://10.15.0.30/api/objects/getbulk/"
        }, {
            "method": "PUT",
            "route": "https://10.15.0.30/api/objects/:objectId"
        }, {
            "method": "PUT",
            "route": "https://10.15.0.30/api/objects/:objectId/properties/"
        }, {
            "method": "DELETE",
            "route": "https://10.15.0.30/api/objects/:objectId"
        }]
    },
    "plugins": [{
        "name": "Speckle Admin",
        "version": "1.0.3",
        "desc": "YEAH! Speckle has a frontend yo!",
        "serveFrom": "/",
        "serveSource": "plugins/SpeckleAdmin/dist",
        "author": "Speckle Project Contributors",
        "contact": "hello@speckle.works",
        "homepage": "https://speckle.works",
        "git": "https://github.com/speckleworks/SpeckleAdmin",
        "sourceDir": "plugins/SpeckleAdmin",
        "canonicalUrl": "https://10.15.0.30/"
    }, {
        "name": "Sample Plugin",
        "version": "0.1.0",
        "desc": "This plugin tests plugins",
        "serveFrom": "/sample-plugin",
        "author": "DAS",
        "contact": "test@dimitrie.org",
        "homepage": "https://dimitrie.org",
        "git": "https://github.com/speckleworks/SpeckleViewer",
        "sourceDir": "plugins/speckle-sample-plugin",
        "canonicalUrl": "https://10.15.0.30/sample-plugin"
    }],
    "jnMask": "######-##"
}
didimitrie commented 4 years ago

Sometimes the debouncer that sends the test request is out of sync, and causes the button to be disabled. If you add a space, or add /api does it work?

If not, the response seems correct, and it probably means there's a problem with self signed certs - I would debug that by capturing the requests with fiddler and seeing if any problems are evident there...

htlcnn commented 4 years ago

Adding a space or /api doesn't work. I'll try using fiddler. Thanks for the hint.

htlcnn commented 4 years ago

This is the first time I used fiddler. SpeckleRevit still fails until I enable HTTPS debugging in fiddle and click Yes in a popup that seems to ask about accepting self-signed certificate (sorry for my bad glance). Now I'm able to add my speckle server. But I still can not add new Sender. It still says Could not access to that server. One thing I notice is that the redirect URL from SpeckleRevit is different from SpeckleAdmin (web app). For SpeckleRevit, redirect URL is always https://localhost:5050. That doesn't exist in my .env. For SpeckleAdmin, redirect URL is https://10.15.0.30/#/signin/callback

didimitrie commented 4 years ago

https://localhost:5050 Are you sure it's https? It should be http. That's where the sign in popup redirects to with the api token when using the account manager - it should be different than the web ui.

Can you post a screenshot of the errors/logs form the console from the revit plugin? To open it click on the ck in the title; it should launch chromium's dev tools. I still believe this is to do with the self signed cert not working properly, and thus requests getting blocked...

htlcnn commented 4 years ago

Yes it was http://localhost:5050. Here's the screenshot from console when I'm not logged in: image

Screenshot after I clicked CREATE SENDER: image

Also a window pops up after I logged in: image

htlcnn commented 4 years ago

This is fiddle's popup. SpeckleRevit's Add New Server button lights up only after I click Yes. image

didimitrie commented 4 years ago

Simplest solution to get stuff going: deploy the server on DigitalOcean on a $5/month VM, you'll get some free credit when you start to keep it going for 5 months i think :)

Otherwise, more complicated answer: seems like there might be something else hogging port 5050: https://stackoverflow.com/questions/29714055/system-net-httplistenerexception-failed-to-listen-on-prefix-http-localhost8

based on the last error...

didimitrie commented 4 years ago

Another quick question (should've asked first!): does it work with the public test server? If so, then I can most definitively point the finger at the self signed SSL cert.

I'll close the issue for now, as it's not a problem with the revit client per se, but let's keep talking here, and maybe we figure your config out...

htlcnn commented 4 years ago

Yes, choosing hestia works.

I have DO instances up and running some production apps. I just want to test on my VM first, and also see if sending data and loading viewer would be faster than using hestia server.

The image saying about port 5050 conflict was my mistake. I added the same server after it was added successfully into SpeckleRevit. I deleted all servers and added it again (still had to keep fiddler on and accept certificate manually).

This is my overall Server config:

map $http_upgrade $connection_upgrade { default upgrade; '' close; }

theoretically this can be a range of severs to facilitate

horizontal scaling.

upstream speckleserver { server 127.0.0.1:3000; }

server {

And: as far as I know websockets are not supported by http2 yet.

listen 443 ssl http2 default_server;
listen [::]:443 ssl http2 default_server;

server_name 10.15.0.30;

include snippets/self-signed.conf;
include snippets/ssl-params.conf;

#dummy certificates, remove after letsencrypt certificate is generated.

#ssl_certificate /etc/letsencrypt/live/speckle.example.com/fullchain.pem;
#ssl_certificate_key /etc/letsencrypt/live/speckle.example.com/privkey.pem;

#ssl_stapling on;
#ssl_stapling_verify on;

# log access + errors
access_log /var/log/nginx/speckle.log combined;

# maintain the .well-known directory alias for renewals
location /.well-known {
    alias /home/speckle/.well-known;
}

# This part might need a bit of tuning later on.
# I'm not 100% sure about the wide range of timeout options available.
location / {
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;
    proxy_http_version 1.1;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $host;
    proxy_pass http://speckleserver;
    proxy_read_timeout 86400;
}

}

- speckleserver .env:

This is a base file for configuring the speckle server. Steps:

2. Copy it to .env ( cp .env-base .env)

2. Configure the .env file with your settings.

3. Start the speckle server!

#

NOTE: If these are set through a different process, eg docker, pm2, or command line

they will not be overwritten.

SERVER_NAME: The server name is important, as it will help users differentiate

between multiple accounts.

SERVER_NAME="BIM UDIC Speckle Server"

CANONICAL_URL: The url address (if you have any) where this server exists. For example,

a canonical url can be "https://hestia.speckle.works".

CANONICAL_URL="http://10.15.0.30:3000"

PUBLIC_STREAMS: Wether all streams created should be public (true) or private (false)

by default.

PUBLIC_STREAMS=false

PLUGIN_DIRS: Dirs to scan for speckle plugins (comma separated). For example, if you install

some other modules in ./test and ./wonderful, you could use PLUGIN_DIRS="./test,./wonderful"

PLUGIN_DIRS="./node_modules/@speckle,./plugins"

PORT: The port you want the speckle server to run on. Make sure the port is accesible

if you have any firewalls set up (ie, ufw).

PORT=3000

IP: The ip to listen to

Defaults to all IPs bound to this machine, and all IPv6 IPs if IPv6 is enabled.

If you want to bind to IPv4 only, or a specific IPv6/v4 address, uncomment the line below and specifiy your preferred address.

IP=127.0.0.1

 MAX_PROC: The maximum amount of service workers to start. If this is not specified,

 Speckle will fork out as many as the machine's processor cores. To specify it,

 just uncomment the line below.

MAX_PROC=1

REQ_SIZE: The request size protects your server from being flooded with too much data.

REQ_SIZE=10mb

This is used in the encryption of the api access tokens.

Please change it to something random!

SESSION_SECRET="xxx"

#

 DBs

#

Mongodb URI: this is the uri string that the server will use to connect to your mongo instance.

 The default string below will work with an out-of-the-box local mongodb deployment.

 For more info, see: https://docs.mongodb.com/manual/reference/connection-string/

MONGODB_URI="mongodb://localhost:27017/speckle"

 Redis URI: this is the uri string that the server will use to connect to your redis instance.

 The default string should work with an out-of-the-box local redis deployment.

 For more info, see http://www.iana.org/assignments/uri-schemes/prov/redis

REDIS_URL="redis://localhost:6379"

#

 Housekeeping

#

If you want your api calls to return pretty json, set this to true. It will cause

the reponses to be ±10% bigger.

INDENT_RESPONSES=false

If you want user email addresses to be publicly visible, set this to true.

EXPOSE_EMAILS=false

The first user to register gets a server admin role.

 A server admin is allowed access to all resources on this server.

FIRST_USER_ADMIN=true

#

Email sending

 This is future functionality that will be used for password resets, email

 confirmation, etc. when using the default auth strategy. It will also be used

 to send notifications (not in the near future though).

#

SMTP server

 This section is used to send emails out. You can use the provider of your choice,

as long as it exposes an SMTP server that you can use. Some of these providers are:

sparkpost, mailgun, sendgrid.

SMTP_HOST="xxx" SMTP_PORT=587 SMPT_USERNAME="xxx@gmail.com" SMPT_PASSWORD="xxx"

 Will populate the from: field in any emails this server sends. Please note, some

 providers will require to verify your domain first.

EMAIL_SENDER="xxx@gmail.com"

#

Authentication Strategies

#

Set to false to disable register routes

PUBLIC_REGISTRATION=true

Local

Local is the default auth strategy for speckle (email + password).

 If this strategy is disabled, the api routes for registering and logging in will not work.

USE_LOCAL=true

 Auth0

 https://auth0.com

USE_AUTH0=false AUTH0_CLIENT_ID="CHANGE_ME" AUTH0_DOMAIN="CHANGE_ME" AUTH0_CLIENT_SECRET="CHANGE_ME"

Azure AD

 https://azure.microsoft.com/en-gb/services/active-directory/

USE_AZUREAD=false AZUREAD_ORG_NAME="CHANGE_ME" AZUREAD_CLIENT_ID="CHANGE_ME" AZUREAD_IDENTITY_METADATA="CHANGE_ME" AZUREAD_CLIENT_SECRET="CHANGE_ME"

 Github

 https://github.com/jaredhanson/passport-github

USE_GITHUB=false GITHUB_CLIENT_ID="CHANGE_ME" GITHUB_CLIENT_SECRET="CHANGE_ME" GITHUB_CALLBACK="CHANGE_ME"

 Whitelisted domains

 Add here any other places that you trust this server to redirect to once authentication is done.

REDIRECT_URLS="http://localhost:5050,https://localhost:8080,https://app.speckle.systems,http://10.15.0.30:3000"

Wether to allow non https redirect urls.

 "localhost" redirect urls will always work with or without TLS.

ALLOW_INSECURE_REDIRECTS=true

#

Telemetry

see: https://discourse.speckle.works/t/community-consultation-time-telemetry/410

This helps us keep track anonymously of the number of speckle servers deployed.

 Disabling this will make Speckle sad :(

# TELEMETRY=true

#

JN Mask

Set a mask to enforce on the job number field in the admin.

Format as: https://v15.vuetifyjs.com/en/components/text-fields#masks

# JNMASK="######-##"

htlcnn commented 4 years ago

Ducking around, I found this issue: Navigating to URL with self-signed certificate shows empty screen

I searched SpeckleUi and found this line: https://github.com/speckleworks/SpeckleUi/blob/05aa7d2480cac21ae56166a81112c6c55158c449/SpeckleUiBase/SpeckleUiWindow.xaml.cs#L53 If SpeckleRevit is using CefSharp < v75.1.141, I think we could add settings.IgnoreCertificateErrors = true; If SpeckleRevit is using CefSharp >= v75.1.141, there's a proposal here: https://stackoverflow.com/a/45077988

Please tell me how could I rebuild SpeckleRevit and test these proposals. Thanks.

didimitrie commented 4 years ago

Speckle Revit is a little PITA when it comes to building and testing. For reference, it's using CefSharp < 75, as Revit comes by default with version 57, so we can't force a newer version from the client side. Make sure that cefsharp 57 is being used! Otherwise things break.

Building and testing it: you'll need to replace the SpeckleUIBase nuget reference in the SpeckleRevit solution with a local clone of SpeckleUi, which you can modify and test.

There's also a quite a few flags that cef can be initialised with, not sure if this will be a better solution - probably not.

htlcnn commented 4 years ago

@didimitrie I tried rebuilding SpeckleUi after adding settings.IgnoreCertificateErrors = true; to SpeckleUiWindow. SpeckleUiTester still Could not access the server. Also, rebuilt SpeckleRevit.dll didn't work, showing a blank page instead. The pain began.

I tried another method and managed to install Let's Encrypt certificate to my server. I created a subdomain and point it (A record) to my Virtual machine's VPN IP (Let's Encrypt cert was issued to my local server successfully using DNS challenge): image

But SpeckleRevit still Could not access that server. Chrome-devtools screenshot: image

htlcnn commented 4 years ago

image It seems that JWT Authorization failed after successful login. I tried to use Postman to send GET request to /api/streams with my token, also got 401. How could I test locally on SpeckleServer's cli?

didimitrie commented 4 years ago

Using something like Postman or Insomnia. There's some instructions here: https://speckle.systems/docs/developers/api-specs

htlcnn commented 4 years ago

I used Postman and used Speckle OpenApi Specs. I edited baseUrl and added my token to Header (key: Authorization, value: JWT... - my token I got from SpeckleAdmin). Response was Unauthorized

htlcnn commented 4 years ago

I must have done something that messed the SpeckleServer up, so it didn't accept some types of request.

The first time, I installed SpeckleServer and setup smtp info. Unfortunately, smtp info was incorrect (I used password instead of app-password for gmail account), so the first register user (also admin user) never received verification email. I thought maybe non-verified user was not allow to create Senders, and I couldn't find resend verification email endpoint in SpeckleServer, so I fixed smtp info and registered another user, login to first user and set 2nd user Admin.

The second time, I removed SpeckleServer folder WITHOUT dropping mongodb database. My efforts weren't succeeded, with all the images above.

The third time, I stop SpeckleServer, drop mongodb database, npm install and run again. Now I'm able to create sender and send data to SpeckleServer.

Thank you very much for your help!

didimitrie commented 4 years ago

No worries @htlcnn! I didn't do much, was more acting like a sounding board. Happy things worked out :)

As a suggestion as to what potentially went wrong when redeploying the speckle server and then getting 401s with existing tokens (ie, not dropping the db): the session secret probably changed (it's in the .env file). That's used to salt passwords and sign the JWT tokens, and if it changes, basically all auth fails!


For the future/if you have time: it would be great to share or summarise your issue on the forum as it's more visible there, and more people can benefit :)

htlcnn commented 4 years ago

I'll make a thread on the forum. Actually I changed secret in .env without thinking it would be related to database. So in that old situation, does registering another new user work? I wonder how I could still login using the web ui, but not via api, I thought they use the same endpoint.

didimitrie commented 4 years ago

I'll make a thread on the forum. Actually I changed secret in .env without thinking it would be related to database. So in that old situation, does registering another new user work? I wonder how I could still login using the web ui, but not via api, I thought they use the same endpoint.

Theoretically yes. Checking the server code, I was wrong: password salting is handled by bcrypt, as it should be. Only JWT tokens for the api are affected.

Logging in the admin works as you get a fresh api token each time. The client side are not regenerated on logins, so removing then adding the account again would have not worked...

htlcnn commented 4 years ago

Forum thread created: https://discourse.speckle.works/t/speckleserver-deployment-notes-on-ubuntu-18-04/532 It's my pleasure to discuss with you. Thanks again @didimitrie