search

vim-scraper / vim-scraper

Scrappy script scraping robot
http://vim-scripts.org/
34 stars 9 forks source link

Revive the vim-scraper/scripts repo #55

Open jdevera opened 13 years ago

jdevera commented 13 years ago

The information in it is invaluable! Perhaps a single JSON file with all info would suffice.

I'm asking because I was writing something that relied on it and only today I read that it would not be updated any more :(

bronson commented 13 years ago

The scripts repo is just a side effect of tracking scraper bugs. Now that the scraper has been burned in, it's not really needed anymore. Not only that, it exposes internals that have already changed.

Yes, I can create a ~20 MB JSON file that includes all scraped data. I'll try to get to this in the next week or two.

(mental note: @bramswenson, author of vimpack, also asked privately about this in March. The existing api files were enough for him)

jdevera commented 12 years ago

I think Ideally we could have a full version and a light version, with the header part for each script only. Something like that could also be leveraged by gmarik/vundle to get script summaries in the search option. This way, each run would not need to get the full 20 MB of JSON. However, for the tool I was writing I'd still need the full version. I think there is value in having both, What do you think?

Edit: I just took a look at the format, what I mean is the light version would exclude the fields description, versions, and install_details.

Edit 2: This filtering would take it down to 537K, which might still be a little too much for each run a BundleSearch

Edit 3: After some further filtering and key compression I get to a total of 300Kb, keeping only the name, type and summary. I know this is very specific, but I guess if these files are going to be generated anyway, something so widely used as vundle is worth having some massaging done for. Maybe I'm wrong.

bronson commented 12 years ago

There are a few json files in /api right now. Check out #14 for some quick descriptions. AFAIK Vundle is already using them for some things (cc @gmarik).

I'm happy to produce whatever files people need. If what's there doesn't fit your needs (and I hear the need for all info, it's coming), tell me and I'll put it there.

jdevera commented 12 years ago

Oh wow, I think scripts_recent covers exactly this last thing I was asking for, thanks for pointing it out.