Cannot select with ViraServers

[chinwobble]

Hey this looks like a really cool plugin.

I've read the entire readme.md but I cannot find how to browse jira issues.

I've created the two json files and tested my username and token are working properly.

I've created the following files according to https://github.com/n0v1c3/vira/issues/41

~/.config/vira/vira_project.json
~/.config/vira/vira_servers.json

I have installed the python dependencies and I have python3 and python in my enviornment paht. I'm using ubuntu 20.04 in WSL

Here's what I have tried:

1. open vim
2. :ViraServers
3. hover over https://bob.atlassian.net/ and press ` (backtick)
4. I get the error E20 mark not set
5. I try press enter but vim hangs for a long time and I end up closing the terminal after several minutes

[n0v1c3]

Hello I am glad to hear you are interested. Logging in is always the most fun one.

What was the backtick for? Have you tried the dev branch?

I just want to clear those up then we can dig in further.

[n0v1c3]

If you are using the __default__ it should auto connect to the server as well.

[n0v1c3]

You should simply be able to not press the backtick. It is looking for a non existent mark. I will add something in to override the error as I can reproduce it just as easily.

For now you should simply be able to not press the backtick. Let me know if that is all it was.

I will keep it open to add a patch in to help prevent it for others.

[n0v1c3]

Also, I assume you are using the cloud service and not a private server? If so make sure you are using the Cloud Password in the readme as you will not use your standard password.

Check also with a raw password entry vs the CMD if there is a difference there it should have been fixed or at least added to my VIRA issues.

[chinwobble]

I'm using dev branch with an atlassian cloud jira account. i.e. company.atlassian..net

for the password I use the token I generated.

I will try out suggestions and get back to you/

[n0v1c3]

The raw password entry will still be the token just don't use the password_cmd mapping just use the raw password mapping. That one will be my best first guess or we will need some actual digging.

[chinwobble]

still not working for me. Here is the full steps:

1. I'm on latest dev branch

2. create ~/.config/vira/vira_servers.json

3. Set the contents to

   {
   "https://company.atlassian.net": {
   "username": "my@company.com",
   "password": "mysecurepassword",
   "skip_cert_verify": true
   }
   }

   I tested these credentials using curl they work. I curl the confluence api using basic authentication https://company.atlassian.net/wiki/rest/api/content and it returns results.

4. open vim and run :ViraServers and it still hangs.

[chinwobble]

Can you tell me which version of the Jira python package you have installed? When I run this in console, it hangs. It might be stuck waiting for user input?

:python3 Vira.api.connect('https://company.atlassian.net')

I have these python packages installed globally

appdirs==1.4.4
black==20.8b1
certifi==2020.12.5
cffi==1.14.4
chardet==4.0.0
cliapp==1.20180812.1
click==7.1.2
cmdtest==0.32+git
cryptography==3.3.1
defusedxml==0.6.0
distlib==0.3.1
filelock==3.0.12
greenlet==0.4.15
idna==2.10
jira==2.0.0
Markdown==3.1.1
msgpack==0.6.2
mypy-extensions==0.4.3
oauthlib==3.1.0
packaging==20.3
pathspec==0.8.1
pbr==5.5.1
pipenv==2020.11.15
pycparser==2.20
Pygments==2.3.1
PyJWT==2.0.1
pynvim==0.4.2
pyparsing==2.4.6
PyYAML==5.3.1
regex==2020.11.13
requests==2.25.1
requests-oauthlib==1.3.0
requests-toolbelt==0.9.1
six==1.14.0
toml==0.10.2
ttystatus==0.38
typed-ast==1.4.2
typing-extensions==3.7.4.3
urllib3==1.26.3
virtualenv==20.4.2
virtualenv-clone==0.5.4

[chinwobble]

@n0v1c3 I figured out the issue. My JIRA cloud account has 12 projects and thousands of fix versions.

https://github.com/n0v1c3/vira/blob/dev/python/Vira/vira_api.py#L826

        # Loop through each project and all versions within
        for p in projects:
            for v in reversed(self.jira.project_versions(p)):
                self.version_percent(p, v)  # Add and update the version list
        return self.versions  # Return the version list

When running :python3 Vira.api.connect("https://company.atlassian.net")

This isn't respecting the filters. So its taking 30 mins to loop through everything.

I set my vira_projects.json to this but it isn't taking any affect

{
  "__default__": {
    "server": "https://company.atlassian.net"
  },
  "vira": {
    "template": "__default__",
    "issueSort": "status",
    "filter": {
      "project": "WEB"
    }
  }
}

[n0v1c3]

what folder are you in? Vira needs to replaced with that folder name for auto filter.

I would start with everything in default or something simple like assignee or in this.

default /server and assignee. I am guessing you do not have millions in your name as assignee.

I will need to relook at it though as we should also have a query limit.

[n0v1c3]

I will also have a good look at this one as I assume your projects have a lot of versions. When this was created we had to go in a back door and not the nice clean filter on the version side like we do on the project side.

[n0v1c3]

Enough digging aside shows that I brock it in VIRA-142.

I know exactly what is going on and this is now the highest priority.

[chinwobble]

Yea it looks like when you call Vira.api.connect() the code hasn't loaded the vira_projects.json file yet.

Is your vira JIRA project public?

[n0v1c3]

Yea the connect is the issue if you comment out the line 183 self.versions = self.getversions() I think this will fix it for you. At least for the initial login.

The real bug is in the function itself.

[n0v1c3]

This is my I paid 10 dollars for account with JIRA and only have a few left out of my 10 so far. I have also open a couple of the free ones and prepared to transfer this one over with their offer although, my server has now been up for 385 days and it will feel not as important to keep my server running every day.

[n0v1c3]

The core of the problem is that I will need to make this into a single query the filters are part of the issue but it is also running a query for each project and version.

Officially VIRA-247: percent complete lag

[n0v1c3]

I did a push on that branch for the initial login test. It looks to me like the bug will exist in the list of versions as well as create an issue. I can easily remove it from create an issue by simply not listing them at the bottom. The menu is the main objective as we do wish to use them in the create issue window as well in the future.

[n0v1c3]

I am also curious about how many users you have roughly and if any improvements will need to be maid there.

[chinwobble]

Can we load versions / users asynchronously?

So I imagine this:

1. type `:ViraIssue` new buffer is created with scaffold of JIRA
2. load the versions / users in the background
3. user types in summary and description
4. After a few seconds the users / versions are finished loaded and the list is populated

[n0v1c3]

https://vi.stackexchange.com/questions/27003/how-to-start-an-async-function-in-vim-8

This is very interesting to me and will create a lot of work for amazing features if it can work.

[n0v1c3]

I have started there and kept on learning and have my first fake async working.

I will write an official push for it and let you know when to give it the first official user trial for me.

[n0v1c3]

@chinwobble Some pushes have started to come to VIRA-247 I need to do some menu front-end clean up and some confirmation that it is not running way too much on the system.

[n0v1c3]

I have got the front-end cleaned I need to do some final testing for the single query at a time then I will pass the testing on to you.

[n0v1c3]

It is now a single query at a time I simply need to clean up the query as there is a mix of projects and versions that will not exist, ie project A has version 1.0.0 and B is only at 0.2.5 A and B will each currently show 1.0.0 and 0.2.5

[chinwobble]

thanks let me know what features you want me to test. I assume it will be in dev branch.

[n0v1c3]

@chinwobble

I am down to "one" bug now. The core of the feature is testable.

Using branch VIRA-247

• Connect to your Jira server
• Quickly open the version filter window ViraFilterVersions
• Have a look and close that window quickly
• Open that window again and have a look and close it
• Once there is an error message at the bottom it is done
• You will have an error message from me that won't go away (working on it)
• Filter Filters are not 100%. (if you have selected a project this will not help yet)

I am most interested in how long it takes for you to get an error message as this tells me how long it takes to query all versions one at a time. How many versions (lines in that menu may be helpful as well)

We can also change a variable in autoload/vira.vim. It is basically taking any spare time to run queries. Once I fix that error message that will only be for the first round. Let me know if you have any CPU affects I did through development but it seems nice to me now.

Thank you very much!

[n0v1c3]

Sorry that variable is s:vira_async_timer = 5

That is in ms but it also must wait to complete and in line for other tasks that are in line. If you have a lot lot of versions there may be some "stuttering" that I did not see even with test at my larger jobs.

[n0v1c3]

OK I have got the core fixed with one more thing to learn for vim to make it amazing. Once I made it work properly I did need to change the timer but this is ONLY for thing like "holding" down a key to do multiple thing ie. I was being lazy and held down j but you will see it jump every 500ms (ish) can you guess what I changed the timer too?

It is not a huge deal and you wont notice it for most however this is a query and when it gets a turn it takes time. That is when we see that vim is truly NOT async!

s:vira_async_timer is still worth playing with as 500 ms may be very slow so I will look into further ways to hide this.

[n0v1c3]

It is now OFFICIALLY working from my prospective. Once you connect you should have a full list quite quickly and then it will get out of the way of vim by slowing itself down for future updates. (I will still need to work on the 'future updates' idea I have).

My work to do.

• Check initial lag with largest server connections possible.
• Ensure future updates on the fly for the project and version menu
• Apply this to the users list I needed to hack for full user details.
• Fix my filter filters

[chinwobble]

This still hangs for me.

1. Checkout branch VIRA-247
2. `:ViraServers` and select my server with lots of versions
3. Press <cr>
4. Can't do anything and i close the terminal after 5 mins.

[chinwobble]

Also this part of the code needs to be guarded. It will throw out of bounds error if the version list is empty

def print_versions(self):
        '''
        Print version list with project filters
        '''

        self.get_versions()

        wordslength = sorted(self.versions, key=len)[-1]

[n0v1c3]

I don't believe it can cause an issue but the real way is to handle all possibilities. I was the rush out the door to at least keep it going. Good thing my years of practice taught me why branches are so good for us.

I believe that the s:vira_asyn_timer variable is the problem I pushed in while testing the code at the end. It is running a query every 1ms. I am looking for the smallest number possible that will not affect your vim usage.

The bigger the number the longer it will take for our list to update. I will also need to add users into this method as the cloud does not give nice names it gives ids so it is another massive query going on.

[n0v1c3]

Start at 1000 and keep cutting in half until you notice it again. That might be are current query speed.

I know I call it async and for vim I am even in shock but it is not truly async so when a query is running nothing is happening. However, when the query is running vim is building a list of what you are telling it to do (ish) and then the OoO is what we are really using for a fake async

[n0v1c3]

There is also a readjustment of that variable once the list is completed once (all projects) that will slow down the queries you will see that in the async function at the top.

All of this is in autoload/vira.vim

[n0v1c3]

Number of users will also require a similar hacking as we are doing the same thing. There is a line in the python connect function for the initial users list. I have temp helped @maricn on that on although you will not be able to use the user menus.

Since it is basically the same issue as this was it is very high priority.

[n0v1c3]

self.users = self.get_users()

[chinwobble]

I'm getting this error because that case isn't handled.

[chinwobble]

Also I did some testing. Since the code isn't really async and it just something to handle timeouts. Would it be easier to do the async logic in python?

import asyncio
async def _get_versions():
    print('started')
    # fetch from the server
    await asyncio.sleep(5)
    print('ending')

async def _run_task_with_timeout(func, timeout):
    try:
        await asyncio.wait_for(func(), timeout)
    except asyncio.TimeoutError:
        print(f"operation timed out after {timeout} seconds")

async def get_versions(): await _run_task_with_timeout(_get_versions, 2)

I tested this logic with :py3 import asyncio; asyncio.run(Vira.main()) and it seems more readable than using viml to handle async. See here. https://docs.python.org/3/library/asyncio-task.html#running-in-threads

[n0v1c3]

Do you have deleted projects?

I very much like going full async into the python side. The variables were all about to move into python anyways as they have no place in vim other than my initial round. All vim should need to do is check a stable variable from python that updates itself amazingly fast I am sure.

Very nice one. Now I have asyncish vim and async python that should be a great combination in the very end of a front end with no async features.

[n0v1c3]

FYI we do have "long long" term plans that vim is just one front end option for the amazing backend python plugin. This helps expand usage for all.

Python will have its core operations with jira code separated completely with the breakdown. Will will then just make our print and vim_print functions.

[n0v1c3]

And of course the queries should have been 100% background. I will save may secret vim async so that your reports can do live updates for you on the screen. After you have a stable connection of course.

[n0v1c3]

@chinwobble I have done a decent push on VIRA-247 it should have the project/version side not holding up your system at all. I do have a few more functions to apply async to but this should make the tool much easy to initially load.

I will be adding the users to that branch as well as I assume that is a similar delay for all larger servers. Once you approve this from your POV I will get it to the dev branch as I know I have more testers there and I will find anything else we should put into this process while in there.

[maricn]

@n0v1c3 Hm, strange, now I get an error trying to execute :ViraFilterProjects:

UPDATE: Actually, whatever :Vira* I try to execute I get that error.

UPDATE2: I tried restarting nvim and running :ViraServers worked to return two lines - my server and Null.. But, still doing <CR> on any, and following up with any command produces the same error.

UPDATE3: My healthcheck is good:

[n0v1c3]

Yes I had just seen this one after my post last night as I was preparing for a dev/master merge with a test.

It works fine in vim but not in nvim... That must be a first to have multitasking in vim and not nvim.

I had a rough idea of where the issue is just not a clue yet what the issue is.

The first couple async functions seem to only be allowed to run once on nvim on my last closer look. I will focus on nvim this evening (Canada time) as once this passes I can spend more time behind the scenes to make it closer to an actual async along with make a big list of everything I want to have async abilities. Ugly tests show these menus and reports will become "live" inside vim.

[n0v1c3]

There is still a non-async function YET that may cause a small "bounce" ever 2 seconds. I am interested in hearing about that from your perspective.

This bounce will be the next attack on the list once nvim passes. It should be a small enough bounce that you will only notice it if you are holding a key down like I do with l and h to test it. I cannot see it with my normal vim usage.

[n0v1c3]

If you comment out the aysnc function and call to it in autoload it should work everywhere but the versions for you.

Edit

I am sure I can stand corrected on that.

[maricn]

If you comment out the aysnc function and call to it in autoload it should work everywhere but the versions for you.

I'm not that comfortable with the code base here, and I've never written a vim plugin myself, but I'll check it out more these days. Now is past my bedtime, so maybe I hear back from you before I get to it. Anyways, thanks for looking into it and sorry for hijacking this issue (i realized late i replied on someone else's issue).

[n0v1c3]

https://github.com/neovim/neovim/issues/3844

Above is the core of the issue. I should be able to avoid this should already not be patched however it is crashing when my async array has blank strings in it... I will do a quick test with a letter stamp in the array as a protector if that works it might just be my work around until I give my mind that break.

[n0v1c3]

https://github.com/neovim/pynvim/issues/342

Even more important and more to come I am sure.

[n0v1c3]

The bounciness you see appears to be an issue directly inside how neovim handles multitasking. Let me know if you can see a difference after I get my next push out here.

The key will be that vim and neovim cannot handle true multitasking. Also neovim appears to be behind with python/asyncio workarounds for code simplicity.

I believe for now this will be the best I am able to do especially with the JQL as the wall I will look closer at neovim and potentially open an issue there about how semi-multi tasking is not handled very well. The vim way of multitasking is essentially the only way it has and I can not tell the difference between python and vim methods for pretending to multitask.