search

e-mission / op-admin-dashboard

An admin/deployer dashboard for the NREL OpenPATH platform
0 stars 9 forks source link

Testing op-admin-dashboard #21

Closed swastis10 closed 1 year ago

swastis10 commented 1 year ago

I have started testing op-admin-dashboard. I checked out dev branch found here.

I loaded data in the dashboard using the script found here. The tar files loaded were -

The dashboard has started to look something like the screenshot attached below.

image
swastis10 commented 1 year ago

Testing Home Tab

Information displayed:

Boxes

Plot

swastis10 commented 1 year ago

Testing Data tab

Tables displayed:

-Export Button This is working well. On UUID, I am able to download csv which has data for all 87 users. On trips tab, I am able to download data for 62735 users.

Errors encountered:

While trying to filter this data using the date filter provided above I encountered some errors. For example: I applied a filter from date 1 Jan to 10 Jan and it did not filter anything and gave the following errors -

Screen Shot 2023-03-16 at 3 43 23 PM

Error resolution:

Started mongodb container again and the filter started working fine.

image
swastis10 commented 1 year ago

Testing Token tab

I tried generating the token which resulted in creating the following error -

Traceback (most recent call last):
  File "/usr/src/app/pages/tokens.py", line 113, in generate_tokens
    insert_many_tokens(tokens)
  File "/usr/src/app/emission/storage/decorations/token_queries.py", line 19, in insert_many_tokens
    insert_many_entries(entry_list)
  File "/usr/src/app/emission/storage/decorations/token_queries.py", line 15, in insert_many_entries
    edb.get_token_db().insert_many(entry_list)
  File "/root/miniconda-4.12.0/envs/emission/lib/python3.7/site-packages/pymongo/collection.py", line 761, in insert_many
    blk.execute(write_concern, session=session)
  File "/root/miniconda-4.12.0/envs/emission/lib/python3.7/site-packages/pymongo/bulk.py", line 528, in execute
    return self.execute_command(generator, write_concern, session)
  File "/root/miniconda-4.12.0/envs/emission/lib/python3.7/site-packages/pymongo/bulk.py", line 358, in execute_command
    with client._tmp_session(session) as s:
  File "/root/miniconda-4.12.0/envs/emission/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/root/miniconda-4.12.0/envs/emission/lib/python3.7/site-packages/pymongo/mongo_client.py", line 1829, in _tmp_session
    s = self._ensure_session(session)
  File "/root/miniconda-4.12.0/envs/emission/lib/python3.7/site-packages/pymongo/mongo_client.py", line 1816, in _ensure_session
    return self.__start_session(True, causal_consistency=False)
  File "/root/miniconda-4.12.0/envs/emission/lib/python3.7/site-packages/pymongo/mongo_client.py", line 1766, in __start_session
    server_session = self._get_server_session()
  File "/root/miniconda-4.12.0/envs/emission/lib/python3.7/site-packages/pymongo/mongo_client.py", line 1802, in _get_server_session
    return self._topology.get_server_session()
  File "/root/miniconda-4.12.0/envs/emission/lib/python3.7/site-packages/pymongo/topology.py", line 488, in get_server_session
    None)
  File "/root/miniconda-4.12.0/envs/emission/lib/python3.7/site-packages/pymongo/topology.py", line 217, in _select_servers_loop
    (self._error_message(selector), timeout, self.description))
pymongo.errors.ServerSelectionTimeoutError: db:27017: [Errno -2] Name or service not known, Timeout: 30s, Topology Description: <TopologyDescription id: 64135f321bca3dd42a198a92, topology_type: Single, servers: [<ServerDescription ('db', 27017) server_type: Unknown, rtt: None, error=AutoReconnect('db:27017: [Errno -2] Name or service not known')>]>

Because no token was generated the exported QRCodes.zip file was empty

Screenshot:

image

Error resolution:

Started mongodb container again and the filter started working fine.

image

Generated QR codes also downloaded fine

If I click on For testing checkbox, the token string generated does contain the substring _test_ in it.

image
swastis10 commented 1 year ago

Testing Maps Tab

We have 3 different map features which are as follows:

Screenshots: Trip lines -

Bubble Heatmap - User_id and email tab remain disabled

image

Density Heatmap - User_id and email tab remain disabled

image
shankari commented 1 year ago

@AlirezaRa94 FYI, in case you want to get a head start on fixing these issues

shankari commented 1 year ago

@swastis10

this error indicates that the mongodb container was not actually up, or you were unable to connect to it.

  File "/root/miniconda-4.12.0/envs/emission/lib/python3.7/site-packages/pymongo/topology.py", line 488, in get_server_session
    None)
  File "/root/miniconda-4.12.0/envs/emission/lib/python3.7/site-packages/pymongo/topology.py", line 217, in _select_servers_loop
    (self._error_message(selector), timeout, self.description))
pymongo.errors.ServerSelectionTimeoutError: db:27017: [Errno -2] Name or service not known, Timeout: 30s, Topology Description: <TopologyDescription id: 64135f321bca3dd42a198a92, topology_type: Single, servers: [<ServerDescription ('db', 27017) server_type: Unknown, rtt: None, error=AutoReconnect('db:27017: [Errno -2] Name or service not known')>]>

Can you verify that the container was running at the time you ran that test?

AlirezaRa94 commented 1 year ago

I guess other errors are related to the database connection as well. Please let me know if this is not the case.

shankari commented 1 year ago

@AlirezaRa94 it looks like the trip table was working, so not sure this is the problem on all tabs

AlirezaRa94 commented 1 year ago

It seems that at the beginning, the connection was proper, but after that, it lost its connection. When you filter data using those date filters, it tries to get data from the database again, and I think the first error is because of the lost connection.

shankari commented 1 year ago

@swastis10 I was able to run the dashboard yesterday without running into the errors that you did, but I only loaded one program. Have you been able to retry with checking whether the database was running correctly?

swastis10 commented 1 year ago

@shankari @AlirezaRa94 all the above mentioned issues got resolved once I restarted the mongodb container

swastis10 commented 1 year ago

Testing Push notification tab

Some use cases of this Push notification tab are :

image
swastis10 commented 1 year ago

Testing settings tab

The settings tab is empty as of now

image
swastis10 commented 1 year ago

Test Dash basic Authentication

As soon as I open the dashboard on Chrome, it prompts me to put username and password.

image

Once, I enter the username and password, the dashboard loads fine.

swastis10 commented 1 year ago

Testing with AWS Cognito

I have changed docker-compose-dash-app.yml to AUTH_TYPE: "cognito" and set config.py using my NREL credentials and I can see the following Cognito login on my screen.

image

I am prompted to create an account with cognito:

image

I enter my email id and password and it sends out a verification code to my email:

image

I also set up an authenticator email:

image

Once done, I am also to see the openpath-stage dashboard:

image

I can also see my details in the user-pool:

image
shankari commented 1 year ago

@swastis10 @AlirezaRa94 All this looks good except for

Once done, I am also to see the openpath-stage dashboard:

Once done, you should see the admin dashboard, not the public dashboard.

swastis10 commented 1 year ago

@swastis10 @AlirezaRa94 All this looks good except for

Once done, I am also to see the openpath-stage dashboard:

Once done, you should see the admin dashboard, not the public dashboard.

@shankari In cognito, when I go to our NREL user pool, I see public dashboard URL as the hosted UI (See attached screenshot). We either need to change the callback to localhost link or need to host admin dashboard on Stage so that we can view that UI once we sign in through cognito.

image
shankari commented 1 year ago

@AlirezaRa94 how did you configure this for your testing? @swastis10 once you work with cloud services to get it to work in the staging environment, we can change the URL to the staging admin URL

AlirezaRa94 commented 1 year ago

@shankari @swastis10 I set the callback URL (redirect URL) to the localhost (http://localhost:8050), both in the user pool and the config.py file.

swastis10 commented 1 year ago

After setting callback URL to http://localhost:8050, I was able to load admin dashboard after Cognito login.

There was another error I encountered:

image

Logs: Traceback (most recent call last): File "/usr/src/app/app_sidebar_collapsible.py", line 212, in display_page raise e File "/usr/src/app/app_sidebar_collapsible.py", line 205, in display_page is_authenticated = authenticate_user(search) File "/usr/src/app/utils/cognito_utils.py", line 82, in authenticate_user user_data = get_tokens(query_params['code']) File "/usr/src/app/utils/cognito_utils.py", line 32, in get_tokens response = requests.post(token_endpoint, data=body, headers=headers) File "/root/miniconda-4.12.0/envs/emission/lib/python3.7/site-packages/requests/api.py", line 115, in post return request("post", url, data=data, json=json, kwargs) File "/root/miniconda-4.12.0/envs/emission/lib/python3.7/site-packages/requests/api.py", line 59, in request return session.request(method=method, url=url, kwargs) File "/root/miniconda-4.12.0/envs/emission/lib/python3.7/site-packages/requests/sessions.py", line 573, in request prep = self.prepare_request(req) File "/root/miniconda-4.12.0/envs/emission/lib/python3.7/site-packages/requests/sessions.py", line 496, in prepare_request hooks=merge_hooks(request.hooks, self.hooks), File "/root/miniconda-4.12.0/envs/emission/lib/python3.7/site-packages/requests/models.py", line 368, in prepare self.prepare_url(url, params) File "/root/miniconda-4.12.0/envs/emission/lib/python3.7/site-packages/requests/models.py", line 440, in prepare_url f"Invalid URL {url!r}: No scheme supplied. " requests.exceptions.MissingSchema: Invalid URL '': No scheme supplied. Perhaps you meant https://?

Another error: Everytime I clicked on the sidebar to go to another tab e.g home, maps, data - It moved me to cognito login screen everytime

AlirezaRa94 commented 1 year ago

@swastis10 See if the TOKEN_ENDPOINT in the config.py file is set correctly. You can also find more information about them in the updated version of the Readme file.

shankari commented 1 year ago

@AlirezaRa94 it looks like I can't regenerate new tokens? I first generated tokens with program = nrelop_smart_commute_ebike_. It generated 50 tokens with the nrelop_nrelop_smart_commute_ebike prefix and displayed them.

Screenshot 2023-03-22 at 8 58 16 AM

I then tried to regenerate with program = smart_commute_ebike but I don't see them displayed. I then tried to regenerated with program = testme for double checking, and I still don't see them displayed

https://user-images.githubusercontent.com/2423263/226965784-f2d9c47a-0d94-4eb9-9f3f-e8717d2032d2.mov

Note that when I export QR codes, I do see them, but they are not visible in the UI.

$ ls -al nrelop_smart_commute_ebike_* | wc -l
     102

There is also no method to remove tokens (in case the admin wanted to change the format, etc).

swastis10 commented 1 year ago

@swastis10 See if the TOKEN_ENDPOINT in the config.py file is set correctly. You can also find more information about them in the updated version of the Readme file.

@AlirezaRa94 I think it is correct. Here are is my config.py, let me know if I need to make some other changes: class CognitoConfig: CLIENT_ID = {clientID} CLIENT_SECRET = '' // this is blank REDIRECT_URL = 'http://localhost:8050' TOKEN_ENDPOINT = '{Cognito Domain Name}/oauth2/token' USER_POOL_ID = {User pool ID} REGION = 'us-west-2' AUTH_URL = '{Cognito Domain Name}/login?response_type=code&client_id={clientID}&redirect_uri=http://localhost:8050'

In Cognito - Allowed callback URLs is set to http://localhost:8050

AlirezaRa94 commented 1 year ago

@shankari The data table in the UI only displays 50 tokens per page and has pagination.

Screenshot 2023-03-23 120823

I couldn't find the method to remove tokens from the database (emission/storage/decorations/token_queries.py). Could you point me to this method?

shankari commented 1 year ago

@AlirezaRa94 I see the pagination now, thanks. you are also right that there does not appear to be a method to delete tokens when this code was originally written by @aguttman, the assumption was that people would create the token list offline and then load it into the database when they were happy with it.

I can think of two possible approaches:

@AlirezaRa94 which is easier for you to implement? I have a mild preference for (1) because it mirrors the workflow for which we designed this, and seems to be easier to implement

We can always tweak later based on admin feedback.

AlirezaRa94 commented 1 year ago

@shankari I prefer the first approach as well.

Another point to mention about the token page is that it displays all the tokens available in the database, including the new ones. It also exports all of them, which I believe is not the intended behavior. We could either show only the new tokens or if we need to include the existing ones, we should separate them.

shankari commented 1 year ago

@AlirezaRa94 I think it would be good to show both but separate them that way, if people forget to hit the "save" button, they are not like "why aren't the new tokens working"? They can see the loaded tokens and understand why. You might want to make the existing tokens page display in reverse cron, though so the newly added tokens will show up first instead of making people paginate all the way to the end.

shankari commented 1 year ago

@AlirezaRa94 production dockerfile does not work; container continually fails start with

Connecting to database URL db
emission/tests/data/real_examples/shankari_2015-07-22
Loading file emission/tests/data/real_examples/shankari_2015-07-22
After registration, test_july_22 -> 28828dff-a670-4c49-878e-96d9f6d95473
Finished loading 0 entries into the usercache and 1906 entries into the timeseries
Traceback (most recent call last):
  File "app_sidebar_collapsible.py", line 20, in <module>
    from utils.db_utils import query_uuids, query_confirmed_trips
ModuleNotFoundError: No module named 'utils'

That's why I ask you to indicate the testing done: https://github.com/e-mission/op-admin-dashboard/pull/7#discussion_r1147762955

AlirezaRa94 commented 1 year ago

@shankari I haven't yet tested the production docker-compose, and it is not ready for your testing. I will let you know when it's done.

swastis10 commented 1 year ago

@AlirezaRa94 , I have created a test App client on NREL AWS console with the following settings:

image image image

In Config.py, my configs are as follows:

class CognitoConfig:
    CLIENT_ID = '5e5c3qdvq6so59alca304f9tv3'
    CLIENT_SECRET = '8htj7mvrnuf34ik6khi7a74qsdvgj11uaja7pmkb4a8oheb1olt'

    REDIRECT_URL = 'http://localhost:8050'
    TOKEN_ENDPOINT = 'https://openpath-stage.auth.us-west-2.amazoncognito.com/oauth2/token'

    USER_POOL_ID = 'us-west-2_O184jYo10'
    REGION = 'us-west-2'
    AUTH_URL = 'https://openpath-stage.auth.us-west-2.amazoncognito.com/login?response_type=code&client_id=5e5c3qdvq6so59alca304f9tv3&redirect_uri=http://localhost:8050'

Even then, I am getting the following error.

image
shankari commented 1 year ago

@swastis10 are there any error logs? For that matter, where do you see logs in a dash app?

AlirezaRa94 commented 1 year ago

@swastis10 I can sign in without any problems with the config that you have provided. Can you check the config.py file inside the docker container and see if it is the same? Is config.py at the root directory of your project?

shankari commented 1 year ago

Closing this for now since portmanteau issues are hard to debug and resolve