Closed swastis10 closed 1 year ago
Information displayed:
Boxes
Plot
Tables displayed:
Participant table: Expected: user ID, labelling stats, first and last PUT and GET calls, trip stats. Available: user_id and user_token
Trip table: Expected - trip attributes, label data. Available - trip attributes, label data
Demographic table: Unavailable
-Export Button This is working well. On UUID, I am able to download csv which has data for all 87 users. On trips tab, I am able to download data for 62735 users.
While trying to filter this data using the date filter provided above I encountered some errors. For example: I applied a filter from date 1 Jan to 10 Jan and it did not filter anything and gave the following errors -
Started mongodb container again and the filter started working fine.
I tried generating the token which resulted in creating the following error -
Traceback (most recent call last):
File "/usr/src/app/pages/tokens.py", line 113, in generate_tokens
insert_many_tokens(tokens)
File "/usr/src/app/emission/storage/decorations/token_queries.py", line 19, in insert_many_tokens
insert_many_entries(entry_list)
File "/usr/src/app/emission/storage/decorations/token_queries.py", line 15, in insert_many_entries
edb.get_token_db().insert_many(entry_list)
File "/root/miniconda-4.12.0/envs/emission/lib/python3.7/site-packages/pymongo/collection.py", line 761, in insert_many
blk.execute(write_concern, session=session)
File "/root/miniconda-4.12.0/envs/emission/lib/python3.7/site-packages/pymongo/bulk.py", line 528, in execute
return self.execute_command(generator, write_concern, session)
File "/root/miniconda-4.12.0/envs/emission/lib/python3.7/site-packages/pymongo/bulk.py", line 358, in execute_command
with client._tmp_session(session) as s:
File "/root/miniconda-4.12.0/envs/emission/lib/python3.7/contextlib.py", line 112, in __enter__
return next(self.gen)
File "/root/miniconda-4.12.0/envs/emission/lib/python3.7/site-packages/pymongo/mongo_client.py", line 1829, in _tmp_session
s = self._ensure_session(session)
File "/root/miniconda-4.12.0/envs/emission/lib/python3.7/site-packages/pymongo/mongo_client.py", line 1816, in _ensure_session
return self.__start_session(True, causal_consistency=False)
File "/root/miniconda-4.12.0/envs/emission/lib/python3.7/site-packages/pymongo/mongo_client.py", line 1766, in __start_session
server_session = self._get_server_session()
File "/root/miniconda-4.12.0/envs/emission/lib/python3.7/site-packages/pymongo/mongo_client.py", line 1802, in _get_server_session
return self._topology.get_server_session()
File "/root/miniconda-4.12.0/envs/emission/lib/python3.7/site-packages/pymongo/topology.py", line 488, in get_server_session
None)
File "/root/miniconda-4.12.0/envs/emission/lib/python3.7/site-packages/pymongo/topology.py", line 217, in _select_servers_loop
(self._error_message(selector), timeout, self.description))
pymongo.errors.ServerSelectionTimeoutError: db:27017: [Errno -2] Name or service not known, Timeout: 30s, Topology Description: <TopologyDescription id: 64135f321bca3dd42a198a92, topology_type: Single, servers: [<ServerDescription ('db', 27017) server_type: Unknown, rtt: None, error=AutoReconnect('db:27017: [Errno -2] Name or service not known')>]>
Because no token was generated the exported QRCodes.zip file was empty
Screenshot:
Started mongodb container again and the filter started working fine.
Generated QR codes also downloaded fine
If I click on For testing
checkbox, the token string generated does contain the substring _test_
in it.
We have 3 different map features which are as follows:
Screenshots: Trip lines -
Bubble Heatmap - User_id and email tab remain disabled
Density Heatmap - User_id and email tab remain disabled
@AlirezaRa94 FYI, in case you want to get a head start on fixing these issues
@swastis10
this error indicates that the mongodb container was not actually up, or you were unable to connect to it.
File "/root/miniconda-4.12.0/envs/emission/lib/python3.7/site-packages/pymongo/topology.py", line 488, in get_server_session
None)
File "/root/miniconda-4.12.0/envs/emission/lib/python3.7/site-packages/pymongo/topology.py", line 217, in _select_servers_loop
(self._error_message(selector), timeout, self.description))
pymongo.errors.ServerSelectionTimeoutError: db:27017: [Errno -2] Name or service not known, Timeout: 30s, Topology Description: <TopologyDescription id: 64135f321bca3dd42a198a92, topology_type: Single, servers: [<ServerDescription ('db', 27017) server_type: Unknown, rtt: None, error=AutoReconnect('db:27017: [Errno -2] Name or service not known')>]>
Can you verify that the container was running at the time you ran that test?
I guess other errors are related to the database connection as well. Please let me know if this is not the case.
@AlirezaRa94 it looks like the trip table was working, so not sure this is the problem on all tabs
It seems that at the beginning, the connection was proper, but after that, it lost its connection. When you filter data using those date filters, it tries to get data from the database again, and I think the first error is because of the lost connection.
@swastis10 I was able to run the dashboard yesterday without running into the errors that you did, but I only loaded one program. Have you been able to retry with checking whether the database was running correctly?
@shankari @AlirezaRa94 all the above mentioned issues got resolved once I restarted the mongodb container
Some use cases of this Push notification tab are :
The settings tab is empty as of now
As soon as I open the dashboard on Chrome, it prompts me to put username and password.
Once, I enter the username and password, the dashboard loads fine.
I have changed docker-compose-dash-app.yml
to AUTH_TYPE: "cognito"
and set config.py
using my NREL credentials and I can see the following Cognito login on my screen.
I am prompted to create an account with cognito:
I enter my email id and password and it sends out a verification code to my email:
I also set up an authenticator email:
Once done, I am also to see the openpath-stage
dashboard:
I can also see my details in the user-pool:
@swastis10 @AlirezaRa94 All this looks good except for
Once done, I am also to see the openpath-stage dashboard:
Once done, you should see the admin dashboard, not the public dashboard.
@swastis10 @AlirezaRa94 All this looks good except for
Once done, I am also to see the openpath-stage dashboard:
Once done, you should see the admin dashboard, not the public dashboard.
@shankari In cognito, when I go to our NREL user pool, I see public dashboard URL as the hosted UI (See attached screenshot). We either need to change the callback to localhost link or need to host admin dashboard on Stage so that we can view that UI once we sign in through cognito.
@AlirezaRa94 how did you configure this for your testing? @swastis10 once you work with cloud services to get it to work in the staging environment, we can change the URL to the staging admin URL
@shankari @swastis10 I set the callback URL (redirect URL) to the localhost (http://localhost:8050), both in the user pool and the config.py file.
After setting callback URL to http://localhost:8050, I was able to load admin dashboard after Cognito login.
There was another error I encountered:
Logs: Traceback (most recent call last): File "/usr/src/app/app_sidebar_collapsible.py", line 212, in display_page raise e File "/usr/src/app/app_sidebar_collapsible.py", line 205, in display_page is_authenticated = authenticate_user(search) File "/usr/src/app/utils/cognito_utils.py", line 82, in authenticate_user user_data = get_tokens(query_params['code']) File "/usr/src/app/utils/cognito_utils.py", line 32, in get_tokens response = requests.post(token_endpoint, data=body, headers=headers) File "/root/miniconda-4.12.0/envs/emission/lib/python3.7/site-packages/requests/api.py", line 115, in post return request("post", url, data=data, json=json, kwargs) File "/root/miniconda-4.12.0/envs/emission/lib/python3.7/site-packages/requests/api.py", line 59, in request return session.request(method=method, url=url, kwargs) File "/root/miniconda-4.12.0/envs/emission/lib/python3.7/site-packages/requests/sessions.py", line 573, in request prep = self.prepare_request(req) File "/root/miniconda-4.12.0/envs/emission/lib/python3.7/site-packages/requests/sessions.py", line 496, in prepare_request hooks=merge_hooks(request.hooks, self.hooks), File "/root/miniconda-4.12.0/envs/emission/lib/python3.7/site-packages/requests/models.py", line 368, in prepare self.prepare_url(url, params) File "/root/miniconda-4.12.0/envs/emission/lib/python3.7/site-packages/requests/models.py", line 440, in prepare_url f"Invalid URL {url!r}: No scheme supplied. " requests.exceptions.MissingSchema: Invalid URL '': No scheme supplied. Perhaps you meant https://?
Another error: Everytime I clicked on the sidebar to go to another tab e.g home, maps, data - It moved me to cognito login screen everytime
@swastis10 See if the TOKEN_ENDPOINT in the config.py file is set correctly. You can also find more information about them in the updated version of the Readme file.
@AlirezaRa94 it looks like I can't regenerate new tokens?
I first generated tokens with program = nrelop_smart_commute_ebike_
. It generated 50 tokens with the nrelop_nrelop_smart_commute_ebike
prefix and displayed them.
I then tried to regenerate with program = smart_commute_ebike
but I don't see them displayed.
I then tried to regenerated with program = testme
for double checking, and I still don't see them displayed
https://user-images.githubusercontent.com/2423263/226965784-f2d9c47a-0d94-4eb9-9f3f-e8717d2032d2.mov
Note that when I export QR codes, I do see them, but they are not visible in the UI.
$ ls -al nrelop_smart_commute_ebike_* | wc -l
102
There is also no method to remove tokens (in case the admin wanted to change the format, etc).
@swastis10 See if the TOKEN_ENDPOINT in the config.py file is set correctly. You can also find more information about them in the updated version of the Readme file.
@AlirezaRa94 I think it is correct. Here are is my config.py, let me know if I need to make some other changes: class CognitoConfig: CLIENT_ID = {clientID} CLIENT_SECRET = '' // this is blank REDIRECT_URL = 'http://localhost:8050' TOKEN_ENDPOINT = '{Cognito Domain Name}/oauth2/token' USER_POOL_ID = {User pool ID} REGION = 'us-west-2' AUTH_URL = '{Cognito Domain Name}/login?response_type=code&client_id={clientID}&redirect_uri=http://localhost:8050'
In Cognito - Allowed callback URLs is set to http://localhost:8050
@shankari The data table in the UI only displays 50 tokens per page and has pagination.
I couldn't find the method to remove tokens from the database (emission/storage/decorations/token_queries.py). Could you point me to this method?
@AlirezaRa94 I see the pagination now, thanks. you are also right that there does not appear to be a method to delete tokens when this code was originally written by @aguttman, the assumption was that people would create the token list offline and then load it into the database when they were happy with it.
I can think of two possible approaches:
@AlirezaRa94 which is easier for you to implement? I have a mild preference for (1) because it mirrors the workflow for which we designed this, and seems to be easier to implement
We can always tweak later based on admin feedback.
@shankari I prefer the first approach as well.
Another point to mention about the token page is that it displays all the tokens available in the database, including the new ones. It also exports all of them, which I believe is not the intended behavior. We could either show only the new tokens or if we need to include the existing ones, we should separate them.
@AlirezaRa94 I think it would be good to show both but separate them that way, if people forget to hit the "save" button, they are not like "why aren't the new tokens working"? They can see the loaded tokens and understand why. You might want to make the existing tokens page display in reverse cron, though so the newly added tokens will show up first instead of making people paginate all the way to the end.
@AlirezaRa94 production dockerfile does not work; container continually fails start with
Connecting to database URL db
emission/tests/data/real_examples/shankari_2015-07-22
Loading file emission/tests/data/real_examples/shankari_2015-07-22
After registration, test_july_22 -> 28828dff-a670-4c49-878e-96d9f6d95473
Finished loading 0 entries into the usercache and 1906 entries into the timeseries
Traceback (most recent call last):
File "app_sidebar_collapsible.py", line 20, in <module>
from utils.db_utils import query_uuids, query_confirmed_trips
ModuleNotFoundError: No module named 'utils'
That's why I ask you to indicate the testing done: https://github.com/e-mission/op-admin-dashboard/pull/7#discussion_r1147762955
@shankari I haven't yet tested the production docker-compose, and it is not ready for your testing. I will let you know when it's done.
@AlirezaRa94 , I have created a test App client on NREL AWS console with the following settings:
In Config.py, my configs are as follows:
class CognitoConfig:
CLIENT_ID = '5e5c3qdvq6so59alca304f9tv3'
CLIENT_SECRET = '8htj7mvrnuf34ik6khi7a74qsdvgj11uaja7pmkb4a8oheb1olt'
REDIRECT_URL = 'http://localhost:8050'
TOKEN_ENDPOINT = 'https://openpath-stage.auth.us-west-2.amazoncognito.com/oauth2/token'
USER_POOL_ID = 'us-west-2_O184jYo10'
REGION = 'us-west-2'
AUTH_URL = 'https://openpath-stage.auth.us-west-2.amazoncognito.com/login?response_type=code&client_id=5e5c3qdvq6so59alca304f9tv3&redirect_uri=http://localhost:8050'
Even then, I am getting the following error.
@swastis10 are there any error logs? For that matter, where do you see logs in a dash app?
@swastis10 I can sign in without any problems with the config that you have provided. Can you check the config.py file inside the docker container and see if it is the same? Is config.py at the root directory of your project?
Closing this for now since portmanteau issues are hard to debug and resolve
I have started testing op-admin-dashboard. I checked out
dev
branch found here.I loaded data in the dashboard using the script found here. The tar files loaded were -
The dashboard has started to look something like the screenshot attached below.