Requirements, Design & Architecture

[suzukimilanpaak]

Requirements

description

Package Indexer

Batch job to index information fetched from Cran web API for all packages.

Implement it with Rails runner where a command to run the task is the following bx rails runner Tasks::CranIndex.revise

• [x] Uncompress PACKAGE and get its DESCRIPTION http://cran.r­project.org/src/contrib/[PACKAGE_NAME]_[PACKAGE_VERSION].tar.gz
• [x] A task runs every day at 12pm to index any new packages
• [x] If new version found, save it as a different record
• [x] Information to be stored and shown in the listing page
  • Packages
  • [x] Package name
  • [x] Version
  • Descriptions
  • [x] Date/Publication
  • [x] Title
  • [x] Description
  • [x] Authors
    • [x] Name
    • [ ] Email
  • [x] Maintainers
    • [x] Name
    • [x] Email
• [x] I don't need to index all the packages, but at least 50 of them

  Package List

Simple web API which responds with packages info in JSON.

• Input GET http://localhost:3000/packages.json
• Output Packages infor in JSON format.
• [x] Create listing page for packages
• [x] Submit its URL for the list as an evidence

[suzukimilanpaak]

Design of Models

• Package
• Description
• Committer
  • type: Author/Maintainer

I used same terminologies as in CRAN API so as not to have unnecessary confusion for another readers of code.

ER diagram

package has_many descriptions and package caches current_description

            ┌────────────────┐
            │                │
            │                │
            │    packages    │──┐
            │                │  │
            │                │  │
            └────────────────┘  │
                     ┬          │
                     │          │
                     │ has_many │ has_one :current_version
                     │          │
                     ▼          │
            ┌────────────────┐  │
            │                │  │
            │  descriptions  │  │
         ┌──│                │◀─┘
         │  │   - version    │
         │  │                │
         │  └────────────────┘     ┌────────────────────────┐
         │           │             │                        │
         │            ─ ─ ─ ─ ─ ─ ▶│committers_descriptions │
         │                         │                        │
         │                         └────────────────────────┘
         │                                      │
 has_many│:through committers_descriptions
         │                                      │
         │           ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─
         │
         │           │
         │
         │           ▼
         │  ┌────────────────┐
         │  │                │
         │  │                │
         └─▶│   committers   │
            │                │
            │                │
            └────────────────┘

[suzukimilanpaak]

Notes & Architecture

Package Indexer

• [x] Any headless browser which can read archived files online?
  • => I could't find any. used linux commands instead
• [x] Careful about speed of the processes not to exceed one day to fetch all info about new package
• [x] Accessing a package page and downloading description are two main causes of time consumption
  • [x] Mult-threading or async job would solve them but let's take the solutions as the next step until the volume of packages and fetching speed does matter
• [x] Save fetched info via ActiveRecord but slice instances of AR into small chunks so that it wouldn't take up RAM

Package List

• [ ] Add index of database tables if needed

[suzukimilanpaak]

Result of Package Indexer

https://www.youtube.com/embed/gyuQvkoiTPE

How long does updating packages take?

It totally depends on mostly how big the size of packages are.

As a tangible example, I sampled first 100 packages;

% time MAX_INDEX_SIZE=100 bx rails runner Tasks::CranIndex.revise
0.67s user 0.11s system 0% cpu 11:00.15 total

So, it takes 11mins to update information for 100 packages.

There are 8302 packages in total. Which means it takes 15 hours to update all packages listed. 15.22 = 11 * (8302 / 100) / 60

But, note that it's the case where update all packages from scratch.

I get the Package Indexer not to access API for description unless package is not updated. ^ Thus, it would takes much less time to update the information for the second and later time. Like, for example, the second attempt 30 mins after the first attempt took 31 seconds as it doesn't have any update of description.

[suzukimilanpaak]

Result Package List

• URL GET http://localhost:3000/packages.json
• Result

[suzukimilanpaak]

Code which might be worth reading

Enumerator.new

sample

I use Enumerator.new quite often for mainly the following two reasons;

• To divide producer phase of enumerable objects and consumer phase of them is good for future me and another coders. It's more readable and they wouldn't mix up the two phases
• Do those two different things in a same loop with lazy evaluation

Actually, a method like in this size needn't utilise Enumerator.new but it's good to use for the purpose of readability and durability for future complex demands.

find_or_initialize_by

I initialised new version of description and used description.new_record? to find if it's needed to be updated. ^

It largely affected shortening time to take Packages Indexer.

has_and_blongs_to_many with STI

description.authors and description.maintainer refer same model Committer.^. This is done by combination of has_and_blongs_to_many and type column of committers table, where author's committers.type = 'Author'. I don't judge it's a good way but it's an experiment.

Other important code in this project

I use normal laws of OOP, including SRP, DI and so on apart from the above.

Model

• https://github.com/suzukimilanpaak/cran_index/blob/master/app/models/package.rb
• https://github.com/suzukimilanpaak/cran_index/blob/master/app/models/description.rb#L3
• https://github.com/suzukimilanpaak/cran_index/blob/master/app/models/maintainer.rb
• and their spec files

Controller

• https://github.com/suzukimilanpaak/cran_index/blob/master/app/controllers/packages_controller.rb
• https://github.com/suzukimilanpaak/cran_index/blob/master/spec/controllers/packages_controller_spec.rb
• https://github.com/suzukimilanpaak/cran_index/blob/master/spec/requests/packages_spec.rb

[suzukimilanpaak]

Thank you very much for reading and your taking time for me.