Open chinwobble opened 3 years ago
Please note https://github.com/pycontribs/jira/issues/622 from our README as well. Not directly related on the connection but worth a look.
I do no not have any major delays though.
I know I broke quite a bit getting it working backwards but it was still very worth it.
Regarding the snakeviz package. Using it to test is fine. We would need to make a good plane to make something like that long term.
We even needed to fold with yaml
and that is just because @mikeboiko and @n0v1c3 like it.
Older Comments
was also broken between the following comets.
| * | 31c6fec VIRA-221: Fixed bug where opening menu/report cleared clipboard
| * | d3bbbbc Fixed dictionary merging bug using deepcopy()
I commented out this code and its fast again.
function! vira#_async() abort "{{{2
" try
" python3 Vira.api._async(Vira.api._async_vim)
" endtry
" if s:vira_async_debug | echo s:versions | endif
call timer_start(s:vira_async_timer, { -> execute('call vira#_async()', '') })
endfunction
I think you can test on your own server by creating hundreds of new versions.
This is the vim profile I get when I just have the vira_report open. This does not include connecting or print_report.
FUNCTIONS SORTED ON SELF TIME
count total (s) self (s) function
8 13.436866 vira#_async()
221 0.081763 0.054034 fugitive#Find()
442 0.050392 0.029758 airline#extensions#ale#get()
221 0.040925 0.027166 airline#check_mode()
1547 0.017236 airline#util#append()
2210 0.015839 0.013823 airline#util#wrap()
221 0.014205 0.012562 <SNR>92_update_hg_branch()
442 0.014639 0.012370 FugitiveGitDir()
667 0.018845 0.012113 airline#util#shorten()
221 0.140207 0.011385 <SNR>92_update_branch()
221 0.093052 0.011289 fugitive#Head()
442 0.011046 airline#extensions#coc#get()
112 0.011688 0.010708 <SNR>74_Highlight_Matching_Pair()
128 0.010266 0.009734 <SNR>56_notify()
221 0.010510 0.009663 airline#extensions#fugitiveline#bufname()
896 0.008813 airline#util#winwidth()
221 0.007909 airline#extensions#keymap#status()
221 0.163484 0.007854 airline#extensions#branch#get_head()
1 0.007846 0.007837 ctrlp#utils#writecache()
121 0.010140 0.007310 ale#cursor#EchoCursorWarningWithDelay()
If you are shutting it right off I would try setting init
and timer
quite high in ms
.
@mikeboiko and I are looking for a way around their extra lag bugs as well.
Also have you tried to comment out the one mentioned above?
I think to implement true async, we need to use the job and channels api in vim. Here is an example I found. https://github.com/OmniSharp/omnisharp-vim/blob/03f8fd4c88d1d5447da55ec499f2ca563e7e54d0/autoload/OmniSharp/proc.vim#L4 https://vimhelp.org/channel.txt.html
Probably smoother and cleaner.
@chinwobble you should also be able to try:
let g:vira_async_timer = 30000
let g:vira_async_timer_init = 30000
That should be 30s apart. Worth a test with tack away the python time.
I just did a quick test, this doesn't block or lag the ui.
let s:save_cpo = &cpoptions
set cpoptions&vim
let s:jobs = get(s:, 'jobs', [])
let s:channels = get(s:, 'channels', [])
function! proc#EchoErr(channel, msg)
let v:errmsg = a:msg
echohl ErrorMsg | echomsg a:msg | echohl None
echom a:channel
endfunction
function! proc#run_job(msg)
let job = job_start(
\ ['python', '-c', 'import time; time.sleep(10); print("hello")'],
\ {'callback': 'proc#EchoErr'})
call add(s:jobs, job)
" let channel = job_getchannel(job)
endfunction
Tested in vim and nvim? (I happen to be off the computer now)
I tested in vim, but not nvim. I think the API in nvim is slightly different.
If you look in the channels help page, there is a reference to an example server. https://github.com/vim/vim/blob/master/runtime/tools/demoserver.py
This has a few advantages over the current approach:
Sometimes we want to block the UI to let users know something is happening.
vim
was fantastic. As always I will need to figure out how to use that for nvim
Might take a bit of work for a clean operation in vim as well.
I have got a small vim
to work however, I may spend my time on another workaround to make sure it works for nvim as well.
This does not directly give me multitasking as the real problem is still JQL and the core solution will still be to create two separate solutions (vim
and vira
) the main goal is to infect vim
with the vira
and not be separated from it. I will also be supporting nvim
all the way as vim
.
BTW: you are able to call directly to vim from that command without an impact witch is pretty good but I want it to be completely unknown.
I will end up back at this issue once nvim
catches up.
I may need to try let s:is_neovim = has( 'nvim' )
No I think the server shouldn't know anything about vim. The server should only handle caching, making api calls to atlassian and reshaping the json to make it easier to work with.
The syntax highlighting, formatting it fields into a table, autocmds, etc should all be handled inside vim outside of hte server.
This way, you can easily test your server logic
We do have a goal for sure to have it out, I guess as long as I think of it as the "n0v1c3 vira" in a single package.
As a user I would like the ability to use this Vim plugin with either Jira or from GitHub to help make it unrequired to even change the software I am using to track my work on different servers. All function Vim level should look the same (unless simply something is only available from one of them) then the functions can do the server type confirmation and changes as required.
VIRA-83 will most likely agree with that.
I basically want to be able to do this from only the single vira window.
@chinwobble I got all the "new" major bugs that I created out of the way. I am sure that the delay is still an issue for you the bugs
just happened to take priority as I tried to start looking at it.
The current dev branch is unusable for me since it keeps blocking the UI.
I'm keen to hear your thoughts on whether using VIM jobs api is the correct the approach.
I have had a chat with @mikeboiko in the real world and we do want this to run as a separate process for sure especially as we keep growing. VIM will purely be the front end in the end.
I do have this as the highest priority still as any "larger" users will want a smooth connection inside vim still. Fun part is just to rewrite all my code 👍😋
I put a temp bypass on the version filter menu I am sure I created some bugs as I step away from my computer for a couple days. This should be the key of the lag you are seeing.
It should still keep a list of the versions you have opened the reports of related issues to.
@chinwobble last push may still be a bit slow for you I did not comment out the full chance that opening a report will add that version to your versions list.
I am sure your versions have thousands of issues in them as this seem to be the core of the problem when I try to recreate it.
I can roll my versions menu way back and it will just be a tag for the version and not longer have the counts for number of issues and number completed. I believe I will also loose my percent complete in the reports.
total = self.jira.version_count_related_issues(idx)['issuesFixedCount']
pending = self.jira.version_count_unresolved_issues(idx)
These lines of code will be the core of my problem. I will work on it "tomorrow" but the issue is that the plugin
I am using goes through all issues hear instead of letting me go through each one individually. I have tested some to make my own copy and that will be the solution. There was a big difference between versions with 1000 issues and 50 issues, even though my base query is for 1 issue.
Update one issue at a time from one version at a time from one project at a time. And, always break away from the ones you were one "last" time. This will make it so we are only truly pulling one issue at a time and slowly updating the percent complete numbers.
I am beginning to play with a db
that is updated slowly and held locally for quicker readings. I will also do another temporary push to the master branch shortly to stop the version updates from lagging your system completely until this is working.
I think it is pretty easy to say now that the db
will be the answer for sure. We can make it handle quite a bit of work however, I will want to make sure it is good and safe and can handle a smooth updating process prior to being released based on the db
. This will now no longer be a complete "re-wright" of the code but quite a large change in operation that was previously planed and additionally enforced.
Thanks @chinwobble! I will keep this one going for sure.
Trying to make a smarter table design with what time I do have to spend. The branch I am on will be dangerous for a bit but eventually the db
will have a version
table that will manage the updates for the design changes. I also need to make sure that it is not just hitting hard on both "developers" side and the "owner's" side so I will make sure that there are some timers, limits and priorities with newer issues to not simply ping over the network all day and all night long as hard as we POSSIBLY can.
I will make a comment below this one to start a list of tables as they start coming to my mind and we may need to discuss some of them as some thoughts are quite large and may need some discussion or at least so on/off flags in my code.
issues
projects
servers
statuses
users
versions
comments??
If we take enough / all of the data it will be possible to have an offline mode where the firs offline step would be that you could at least read all the data from vira
while you are disconnected. Going further we would want to hold comments for when you are back online, much much more fun.
I hope that patch VIRA 0.4.8
will help out well I work on a real longer term solution. This should also help with the speed of the report loading time if that was a concern at all.
[ ] Percents Removed
[ ] Version List Broken
The base of a db
is coming to life on my branch VIRA-253 it still doesn't really do anything useful but all the features are their for some real functionality to start happening. The first round of the db
wont be to bad as I will not worry if it will need to loose its data for next updates however version handling and proper patching will be a high priority as well inside this issue as it should be with db related ones.
I am going to start with a simple state tracker for all issues on the servers you are connected to. It will most likely start with a very simple straight line through the issue numbers for updates but I already have some notes in my real issue for some better logic there as well.
currentUser
My current goal is to keep reducing the async time and simply using logic as we should. Vim with async, db and a network will now basically be able to do anything so also please keep telling me how to improve with ideas like this one. :) - I will block that bad issue mapping one of these days! (VIRA-252)
I can you see you put a lot of working into working on the db
.
Do you think it will be better to split all the db related code into a different python class.
This vira_api.py
file is getting pretty large now.
Good day @chinwobble, I have also told myself that as the db code keeps growing, I just had it in there to get the core of the db running and I am basically there now.
You should be able to have a test run if you are interested, the only thing is no db version control exist yet as it is next on the list. It should just update/delete your first db/tables once that is in place to control my table changes for users.
I am also becoming more and more interested in store basically all the data we display in properly designed tables. This will take up some local storage space however, our original async will now wast almost 0 network and become very very fast as it will be looking at the local db on searches. A query to the JQL server can be closer together only for recent updates if
there was on check again shortly if not take a longer break and check for updates issues. This will become an important step to make us wast very little "data".
If you are willing to have a test you will see that you have, well do not have data in your database when you first start. The following commands have not been added into the code mappings/logic but you can add them into .vimrc
.
" One second updates ( "vira test slow" )
nnoremap <silent> <leader>vts :let g:vira_async_timer=1000<cr>
" Basically as fast as the network will let you updates ( "vira test fast" )
nnoremap <silent> <leader>vtf :let g:vira_async_timer=25<cr>
A file will also be created in ~/.config/vira
called vira.db
I will need mapping for actual locations there as well but, I will be interested in the size of your file at this current point and how many issues it has found (this is the core of the bugs I am looking at and it is mostly my sloppy query style I hope)
Currently only works with one server, looking into that next. I am also going to slow it down with an option for the fast update.
I have also caught on the side that issues can be missed randomly and a list will need to be created of "possible" missing issues. I know they can be transferred to projects and still apparently show up in the original project. Work around it will need to be made for all cases and make sure it is no longer tracked as missing once found, even if the label dosn't make sense.
@n0v1c3 if you want to test with a large jira server, maybe you can sign up to apache's jira and query their api. https://issues.apache.org/jira
@chinwobble Just added them to the list. I will add them to the tests for the db.
May "guess" is it can handle it. :) with latest pushes.
@chinwobble welcome to the GitHub team! I am making the plan on Jira along with the clean merge with GitHub issues back and forth.
I have at least created an epic for now VIRA-279. You can use this for branches and labels until it gets even better.
I try to use the right commands to label my pushes with the issue as a prefix ie "VIRA-279: ..." this makes GitHub look nice.
As for the dev once you are happy with something you are working on simply merge it into there (also with the issue label "VIRA-279: ...").
My version 0.5.0 goal was the epic issues and some other key features. I put it on hold as the db is changing everything.
That was the correct website to do the testing on BTW. I got to GERONIMO-1632 and then I could not make it any further.
This happened to be the worst issue to crash on as it was called "Test" and I happened to crash there but it also happens to be the last of the more than one thousand issue that were closed on 2006/08/07 (August). The first official day of more than 1000 Jira updates on https://issues.apache.org/jira
This is a bug with my updating by timeline and no handle for more than 1000. There are a few ways I should be able to make this work with the best looking one for now down bellow. I will have a look at this next round and should be able to get past it.
I will have more to clean it up but it looks like that should have got the query.
And, now a cleaner, even faster query. My last issue there is simply the query time when pulling from https://issues.apache.org/jira is simply slower than my server in my basement. It will currently take me about a week to update my database with all of their information using JQL
especially inside vim
.
I am going to finally focus on make the things I have broken work, mostly menus now. Then I will most likely replace all features of all menus along with the reports from the db to call this done.
Outside of vim
will be an issue all of it's own after this one and Epics
are cleaned up for VIRA 0.5.0
release requirements. I was in the middle of Epic cleanup until db got on my mind.
db
now has it's own class ViraDB
and it will be cleaned up more and more. I did not worry about braking the front end as the DB is working.
As a user, I don't want an open VIRA report to be slow and I don't want each report to take a long time to load.
I suspect something is wrong with the async code.
I'm going to put a few examples of how to profile the python and vim code.
tabs
win....vira_prompt
is open on the fly and we need strings and numbers to work properlyreport
goes blank (VIRA-213) [627b6c4]profile python code
Then i use snakeviz to check the results.