search

danielchalef / mrfparse

A Go parser for Transparency in Coverage MRF files.
Apache License 2.0
17 stars 8 forks source link

Downloader doesn't handle query strings #25

Open cigzigwon opened 11 months ago

cigzigwon commented 11 months ago

When trying to access files on a webserver it doesn't work: https://bcbsil.mrf.bcbs.com/2023-08_361_50I0_in-network-rates.json.gz?&Expires=1695132051&Signature=KLRMAB4vlXMcDEuTocPFwR00lEMbnzuGBNGeTcTcjyPBQVhGpUxx-CIQu6ePzTRDDvS7ZYroay-YDoeU3TaMcKl51m5OjF9BBzL2V33Kd8jXIcOZMU3fblygJcp4sr2359Ec05zdcKO6ecsHFNiV~lvGcNCezoNOIkzFR0aR2hB~VPas31Sfdplp6NiBgqflKzqBBaR~bbOLrb3U3thCE6ffSMI3r8xA4EbMWlxTE1fqnfLse-Y-tWL1V7SD7t3CFlLP~fKg~sCiqbyCMOLklfIg-R3HfUsl9r0RRpU9gvA3OCec9nQyOGmtipciVKJ-RTB-CS4HeiafXPojoSch8A__&Key-Pair-Id=K27TQMT39R1C8A

cigzigwon commented 11 months ago

GL-MO-LT-00004:bin csobeck$ docker run -it -v "$(pwd)/tmp/mrf-parse:/tmp/mrf-parse:rw" dancarbone/danielchalef-mrfparse pipeline -i "https://bcbsil.mrf.bcbs.com/2023-08_361_50I0_in-network-rates.json.gz?&Expires=1695132051&Signature=KLRMAB4vlXMcDEuTocPFwR00lEMbnzuGBNGeTcTcjyPBQVhGpUxx-CIQu6ePzTRDDvS7ZYroay-YDoeU3TaMcKl51m5OjF9BBzL2V33Kd8jXIcOZMU3fblygJcp4sr2359Ec05zdcKO6ecsHFNiV~lvGcNCezoNOIkzFR0aR2hB~VPas31Sfdplp6NiBgqflKzqBBaR~bbOLrb3U3thCE6ffSMI3r8xA4EbMWlxTE1fqnfLse-Y-tWL1V7SD7t3CFlLP~fKg~sCiqbyCMOLklfIg-R3HfUsl9r0RRpU9gvA3OCec9nQyOGmtipciVKJ-RTB-CS4HeiafXPojoSch8A__&Key-Pair-Id=K27TQMT39R1C8A" -o /tmp/mrf-parse/outputs -p -1 INFO[2023-08-06T20:40:56Z] Running step: Download ERRO[2023-08-06T20:40:57Z] Fatal error in /app/pkg/mrfparse/pipeline/steps.go#116: open /tmp/mrfparse3808262212/src/2023-08_361_50I0_in-network-rates.json.gz?&Expires=1695132051&Signature=KLRMAB4vlXMcDEuTocPFwR00lEMbnzuGBNGeTcTcjyPBQVhGpUxx-CIQu6ePzTRDDvS7ZYroay-YDoeU3TaMcKl51m5OjF9BBzL2V33Kd8jXIcOZMU3fblygJcp4sr2359Ec05zdcKO6ecsHFNiV~lvGcNCezoNOIkzFR0aR2hB~VPas31Sfdplp6NiBgqflKzqBBaR~bbOLrb3U3thCE6ffSMI3r8xA4EbMWlxTE1fqnfLse-Y-tWL1V7SD7t3CFlLP~fKg~sCiqbyCMOLklfIg-R3HfUsl9r0RRpU9gvA3OCec9nQyOGmtipciVKJ-RTB-CS4HeiafXPojoSch8A__&Key-Pair-Id=K27TQMT39R1C8A: file name too long

cigzigwon commented 11 months ago

This is moot since I can wield mrfpare split and mrfparse parse in my current pipeline "fetch" stage. It just becomes a file to process locally.

I appreciate the work done here that deals with extraneous providers. At this point we are gathering all billing codes.

Is it possible to disable the services code filter CSV?

danielchalef commented 11 months ago

You should be able to override the Service List filter here: https://github.com/danielchalef/mrfparse/blob/1781ef154e0647e7ee05043398379fbb678cb288/pkg/mrfparse/mrf/in_network_rates.go#L404

Note that it's been a long time since I looked at the code and the MRF files were something of a moving target back in December. You may unfortunately need to dig in to make sure the parser still functions correctly and does so for all payers.