Article: Searching internet RFCs

ghost commented 4 years ago

Hi, I want to start my "Open source knowledge" sharing here, I will pull 4-5 articles that I wrote last months!

briandfoy commented 4 years ago

Great! Let's start with one article as part of this pull request.

ghost commented 4 years ago

ok, this is the first time that I use github so I hope to do it correct!

ghost commented 4 years ago

`{ "categories" : "tooling", "description" : "Smart ways for scraping RFCs", "draft" : true, "date" : "2020-18-16T07:28:49", "authors" : [ "Edoardo-Mantovani" ], "title" : " an automatic perl-based RFC downloader ", "tags" : [ "Request for Comments", "perl tools","protocols knoweledge" ] }

Since my first days spent on studying computer security the concept of "protocol" fascinated me, maybe for their enormous diffusion in almost every computer system, we can say that our day lives heavily depends from these optimized alghoritms and as I say "trust on machines but don't trust on humans", I find interesting as starting point the gathering of these documents for understand why don't trust on us. During the quarantine I was able to find the the good side of the home confination: I hadn't enough time to read a book due to school's tests, but for my luck, I had enough time for reading one or two RFC documents. I find complex to to browse the www.ietf.org for searching those documents, so I made a fast and efficent script which permit to download RFCs through a keyword and display on terminal for decide which one can be saved and which can be deleted.

what are RFCs?

The most important point in this document is to define the "Request For Comments" uses and philosophy, infact I think that one of the best online references for learning how networking really works ( at protocol level )are these simply documents ( stored as text file) which defines every single standard that has been approved by the Internet Engineering Task Force, these documents are a good entry point for developing the study of the protocols analysis: an emerging field which uses formal verification for trying to find exploits. One thing that we must know is that every RFC has a title and its associate number, for example the rfc number 3607 is the Chinese Lottery Cryptanalysis Revisited: The Internet as a Codebreaking Tool. The RFC approach reminds the open source philosophy school, which has the same objective ( give everyone the opportunity to learn new things through the code shared ) and the same propagation channel: the internet.

The script will use the [Net::RFC::Search]({{<cpan "Net::RFC::Search">}}) module as back end for the search, and the [Term::ANSIColor]({{<cpan "Term::ANSIColor">}}) module for a pretty print displaying ( there will be a lot of text in the terminal and different coulrs can help! ).

The script is as follows:


#!/usr/bin/perl 
# Made by Edoardo Mantovani in 2020

use strict;
use Term::ANSIColor;
use Net::RFC::Search; 
use File::Path 'rmtree';

my $folder = shift or die "./RFC <Folder> <Query>\n";
my $query  = shift or die "./RFC <Folder> <Query>\n";
my $rfc_interface = Net::RFC::Search->new();
my @rfc_array = $rfc_interface->search_by_header( $query );
foreach ( @rfc_array ){
  my $Ffolder = $folder . $_; # i.e final folder = /tmp/1110 
  if ( $rfc_interface->get_by_index( $_, $Ffolder ) ){
    print colored("Downloaded: $_ \n", "green" );
    &view_rfc( $Ffolder );
  }else{
    print "Error with RFC $_\n";
}

  }

sub view_rfc( ){

  print `head -n 25 @_`; #show only a small preview
  sleep(2);
  print colored("Do you want to save this RFC? [Y/N] ", "red");
  my $input = <STDIN>;
  chop( $input ); # remove the 'return' character from the str
  rmtree(@_ ) if ( $input eq "N" );

  }

let's study some interesting piece:

$rfc_interface->search_by_header( $query );

search from the ietf.org the header contained in the $query variable and return the associated numbers, these are saved in the __@rfc_array__.

$rfc_interface->get_by_index( $_, $Ffolder );

the get_by_index function accept as first parameter the number associated to the RFC, and as second parameter the directory for saving the file.


sub view_rfc(  ){

  print `head -n 25 @_`; #show only a small preview
  sleep(2);
  print colored("Do you want to save this RFC? [Y/N] ", "red");
  my $input = <STDIN>;
  chop( $input ); # remove the return character from the str
  rmtree (@_) if ( $input eq "N" );
  }

the __view_rfc function can be considered our "visualization" subroutine, which allow us to read the first 25 lines from the text file and ask if we, after the preview, want to save the file or delete it and pass to the next, Note_ that the @ variable contain the argoument passed to the function, in this chase is the $Ffolder variable ( final folder ).

other uses and suggestions I made this simple script mainly for save time and have a better interface for interacting with the RFC site, I want to recommend to see the TODO part in the [Net::RFC::Search]({{<cpan "Net::RFC::Search">}}) page, for a better script, it would be cool to implement curl and lynx for retrieve cancelled RFC's, of course the script can be implemented with custom functions but this is script is only for educational purposes.

resources

RFC's story: https://blogs.helsinki.fi/mildred/2017/01/31/request-for-comments-user-stories-and-scenarios/
Other RFC's informations: https://en.wikipedia.org/wiki/Request_for_Comments

`

briandfoy commented 4 years ago

You have a branch for this pull request already:Baseband-processor:patch-1. Commit to that branch and whatever you do will become part of this pull request.

briandfoy commented 4 years ago

This article is a very good start, but there are some things to beef up.

[x] First, that rm @_ is not a god idea. Look at File::Path's rmtree instead.
[x] Also, you don't need to shell out to head. Perl can do that for you.
[x] You might like colored( [ ... ], ... ) so you don't have to do those resets.
[ ] Can you explain Net::RFC::Search a bit more? Show some different sorts of queries.
[ ] Do you have a personal story about why this is interesting for you? Tell us what got you started on such a thing.
[ ] Pick out a particular RFC and tell us why you want to look at it
[ ] Some of the RFCs aren't all that small ;)

ghost commented 4 years ago

For the Head do you think should be good to open the Downloaded file and read the first X lines? I think that this could "speed down" the script.

Otherwise thanks for your suggestions, I have made some enhancement :)

ghost commented 4 years ago

Hi, I have written other articles about "accademic documents scraping" and SDR in perl, can i open a pull request or I must wait for the finish of this article?

briandfoy commented 4 years ago

That sounds like an interesting idea. Add an issue instead of a pull request right now. Let's get you through your first article so you have a better idea what the next articles should look like. Remember, we're just volunteers doing this in our free time, so don't dump a bunch of work on us all at once. :)

ghost commented 4 years ago

ahhahah Ok, don't worry, I'll try to don't spam my articles!

ghost commented 4 years ago

Hi Brian, I wrote an article about wireless security in Perl and I have also made a quite simple script which is similar to "aircrack-ng", I have only one question: How deep I can explain things? should I have a limit ? ( the script can "hack" wireless network and, personally, I don't like lamer-articles ). for example, in the article, can I post the code which execute the file extraction inside a network or I it would be better to skip and show only some basics functionalities of my script? I have made also a list with suggested modules and other possible uses. (the link about data extraction: https://www.sans.org/reading-room/whitepapers/tools/extracting-files-network-packet-captures-36562) Thnaks.

Edoardo

briandfoy commented 4 years ago

One article at a time, but if you have another article idea, you can make a GitHub issue with it. That why we don't lose track of it in the comments for an unrelated article. If you'd like to discuss your articles, reddit.com/r/perl or www.perlmonks.org might be good spots.

ghost commented 4 years ago

Hello, could I know if my article has been accepted? Regards.

briandfoy commented 4 years ago

I see that you put a new version of the article in the comments here. Can you commit that to your branch? The pull request will update automatically.

ghost commented 4 years ago

how can i commit it into my branch? sorry but i am not practical on github :)

briandfoy commented 4 years ago

Okay, I'll take care of those details. The article looks like a good starting point, but I need to do some work on it. When I get some time to pay attention to Perl.com, I'll commit your new version and fix up the code.

ghost commented 4 years ago

Ok thanks

ghost commented 4 years ago

Hi, when it will be published?

thanks.

briandfoy commented 4 years ago

This isn't close to being published. I'm having trouble getting the code to work. For some reason, my version of Net::RFC::Search can't fetch the index file. And there's that list of things I noted earlier that we need to fix.

briandfoy commented 4 years ago

Okay, one of the issues is an LWP problem. The HTTPS protocol was kicked out to its own distribution, LWP::Protocol::https. People need to install this before they use this module. The article will have to address this since the Net::RFC::Search module hasn't bee updated in 7 years and that's the last time the module author was active on CPAN.

ghost commented 4 years ago

Strange, on my computer everything works fine. Yes the module is a bit old, but the RFC are much older :)

briandfoy commented 4 years ago

"Works for me" isn't something you want to say. When you are presenting code to other people, especially in an article, you want to ensure that you explain everything they need. For this, I often use a fresh Perl to try things.

ghost commented 4 years ago

which operating system are you using?

I tested the script in a fresh Ubuntu system and, after the "cpanm Net::RFC::Search" everything works fine.. in general the Ubuntu (and Debian) based distros are the most popular and, specially Ubuntu has a good repository for perl modules, if I must test the script also for other os there is no problem.

briandfoy commented 4 years ago

I'm only speaking in general about "works for me". I've solved the problem now. It's that there's an undeclared dependency. It's not an OS problem. Again, the point is that you have to understand everything that goes into your instructions, so think about that when you write future articles.

ghost commented 4 years ago

Okay, I'll follow your advice next time :)

Il giorno ven 31 lug 2020 alle ore 00:57 brian d foy < notifications@github.com> ha scritto:

I'm only speaking in general about "works for me". I've solved the problem now. It's that there's an undeclared dependency. It's not an OS problem. Again, the point is that you have to understand everything that goes into your instructions, so think about that when you write future articles.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/tpf/perldotcom/pull/248#issuecomment-666761278, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQHT2IXJLLDSF3QKWSJY7D3R6H3EJANCNFSM4OWOPK3A .

ghost commented 4 years ago

Hi Brian,

I don't know why, but my article is still unpublished, I don't care much on it, I have written even more interesting articlesrelated to wireless security and sdr. in those week I've written also the Air::Lorcon2 library, (link: https://github.com/Baseband-processor/Air-Lorcon2). Now, I would like to publish an introductive article on my library, which is quite big (more than 200 functions, some implemented in pure-perl), Could I know if I can open a pull request in the next few days for the article request?

Regards Edoardo Mantovani, 2020

note: the tests still crash, I need some days for fixing this

briandfoy commented 4 years ago

I've been away from the computer and I still need to do some editing work for your article.

I think that you should wait for any new articles to see the changes that we make and to work on what you've already done to ensure things are correct. The more work we as editors have to do, the longer things take. We don't have a lot of free time, so introducing extra work slows us down even more.

Additionally, we wouldn't publish an article about a library with failing tests.

perladvent / perldotcom

Article: Searching internet RFCs #248