kym834
commented
4 years ago Hi @Ecaloota,
Looks like it has been very successfully, not moderately! Comments/thoughts below just based on the order of what you have written above.
-
All good on EBS, like you said same source.
-
British vs. American english at play with aciclovir/acyclovir I think. Something we need to keep an eye on as we expand to other countries and with other medicines as well.
-
Is there a way to remove the random multivitamins without having to do it manually? For some context, the list we have been working with only includes teh anti infections medicines that are on the WHO essential medicines list. In future, we want to expand to the whole WHO list which is much bigger and thus this might be quite time consuming.
-
The other combination medicines may be in other sections of the WHO's list so, you they may be important in that regard. Especially in later stages when we expand to the whole list.
-
Yup, we didn't find any of that information on the ATGR website either. We will have to see if we can find any of this information elsewhere. Also it may be possible that the patent information we collected for the USA are also relevant for Australia if the patent in worldwide. Again something we will have to look into further. I have a vague memory of a status but I think it was register status and it was in the product information pdfs. I went back and had a look through a few pdfs for different medicine and couldn't find it. It may be that this is not consistent across all the information documents as with the approval dates.
-
We'll have a think about how we might be able to identify generics. Maybe this information is held somewhere else or we just need a new criteria.
-
Approval is somewhat redundant. With automation if we don't get info for it, highly likely that it is not approved. We had that in when we were manually searching for the information and had to check each one individually. It would useful for a separate data set that could be used for comparisons of which medicines are approved and which aren't in different countries. But I think this would be a separate excel file to this one.
-
You did them all manually! I hope that didn't take too long.
-
Output files look good :)
I'll have a deeper look at the output file and mull over some of the things that have been brought up over the weekend so that we can discuss more in the meeting next Monday (#9)
Thanks so much for doing this work and the update!
Ecaloota
Hi all,
Following our meeting on the 13th July, I would like to update you all on my progress with writing a script to automate data extraction from the ARTG, which has been moderately successful.
I used the database at the EBS to extract a list of all the potentially-relevant ARTG IDs (I didn't use this one, because it doesn't allow logical operators, but the two databases say they are linked to the same source) by searching for all drugs that were of interest in Stage I (I added "aciclovir" myself as it appears to be the more-common spelling of "acyclovir"). This had to be done in seven steps due to limitations with the Aus Gov servers, so duplicate ARTG IDs introduced at this stage were removed manually.
I then wrote a bot to search for all the drugs by ARTG ID on the latter database. Most of the available information is on this webpage, so I extracted the product names, strengths, IDs, dosage forms, routes of administration, and sponsor. If the entry had an associated Product Information PDF file (apparently only 90% have one, but I think it's probably less than that), I downloaded that automatically and converted it to a plain-text file. At this stage, it became clear that 530 of the entries I'd collected were of random multivitamins that were included due to ambiguities in the search (my fault), so I removed those manually in Excel. The thus-cleaned list of ARTG IDs is here.
Looking through the remaining entries, I also found some combination drugs that were comprised entirely of drugs that were individually in our search, but not found in that particular combination. For example "abacavir + lamivudine + zidovudine". I kept those in the cleaned list because I considered that they could be important also. The list of ARTG IDs for these ambiguous combination drugs is here.
Of the data that remained to be extracted (under patent/patent date, exclusivity/exclusivity date, market status, generic forms), I don't think the ARTG stores the data for these, which makes determining whether one drug is a generic form of another impossible on the basis of the criteria we used in for Stage I. Also, I believe that collecting information on approval (y/n) might be redundant in this case, as only the approved drugs seem to appear on the database? Happy to add this to the data if I'm wrong, of course.
That only leaves approval dates, which are the only piece of data in the PDFs that can't be extracted from the web-server. So, because of very inconsistent formatting of both the PDF files and the product name conventions, I did both manually for the remaining ~1500 entries based on what I had already collected into the spreadsheet. The output of my script following manual cleanup is here, (and prior to manual cleanup is here).
The Python script and bot itself is here. I would not recommend running it if you have limited data from your internet provider because it will try to automatically download thousands of PDF files, and has many dependencies. I can also send you all the downloaded PDFs, but there are too many to be uploaded here.
As always, it would be great to have everything double-checked, and I can make adjustments if required in about a week and a half.
@yaelago @kym834 @alintheopen @fantasy121